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1.0  INTRODUCTION 


At  the  request  of  the  Software  Technology  for  Adaptable  Reliable  Systems  (STARS)  Joint 
Program  Office  (SJPO),  the  Institute  for  Defense  Analyses  (IDA)  undertook  a  review  of  the  STARS 
Program  during  September  and  October  of  1985.  Three  concerns  prompted  the  review:  (1)  the 
Program  is  to  receive  $42  million  in  FY86,  (2)  the  perception  that  STARS  is  not  effective,  and  (3) 
the  absence  of  an  effective  top-down  plan  for  the  Program. 

The  following  review  panel— consisting  of  IDA  staff  and  consultants— conducted  the  review: 

Dr.  Elizabeth  Bailey  (Software  Metrics  Inc.) 

Dr.  Richard  DeMillo,  Georgia  Institute  of  Technology 
Mr.  Herman  Fischer  (Mark  V  Business  Systems) 

Ms.  Audrey  A.  Hook  (IDA) 

Dr.  John  F.  Kramer  (IDA),  Chairman 
Dr.  Thomas  H.  Probert  (IDA) 

Mr.  Samuel  T.  Redwine  (IDA) 

Dr.  William  Riddle  (Software  Design  &  Analysis,  Inc.) 

Dr.  Robert  I.  Winner  (IDA) 

Some  of  the  reviewers  have  worked  on  IDA  tasks  for  the  STARS  Program  for  several  years;  others 
have  consulted  occasionally  on  STARS;  and  a  few  had  no  prior  exposure  to  the  Program.  This  mix 
helped  provide  a  more  balanced  perspective  on  the  subject 

Another  group  was  briefed  on  the  progress  and  findings  of  the  panel  at  a  meeting  on 
September  26,  1985;  this  group  consisted  of  the  following  members: 

Mr.  Joseph  Batz 
Dr.  Barry  Boehm 
Mr.  Bill  Carlson 
Dr.  Neil  Eastman 
Mr.  Joseph  Fox 
Dr.  Ugo  Gagliardi 
Dr.  Leonard  Haynes 
Dr.  Ed  Lieblein 
Dr.  Edith  Martin 

Many  of  these  reviewers  were  subsequently  contacted  on  an  individual  basis  regarding  specific  areas 
of  concern.  In  addition,  the  panel's  preliminary  conclusions  were  briefed  to  the  Defense  Science 
Board  panel  headed  by  Fred  Brooks  on  October  23,  1985.  The  comments  and  advice  received 
during  these  briefings  were  quite  helpful  in  shaping  the  panel's  findings. 

This  review  proceeded  in  several  stages: 

(1)  The  first  stage  was  a  discussion  and  analysis  of  the  origins  of  the  STARS 
Program  and  the  events  leading  up  to  its  current  state. 

(2)  The  panel  then  attempted  to  understand  the  internal  and  external  organizations  and 

activities  relevant  to  the  STARS  Program.  This  included  both  the  technical  and  political 
aspects  of  these  organizations,  as  well  as  their  inter-relationships. 

(3)  The  panel  then  assessed  the  most  recent  STARS  plan  (19  September  1985)  [1].  The 
assessment  concentrated  on: 


a.  The  Program's  goals  expressed  in  the  plan 

b.  The  earlier  goals  as  analyzed  in  the  STARS  Goals  and  Objectives  Working 
Group  report  [5] 

c.  The  goals  for  the  six  STARS  organizational  areas  (as  stated  in  the  plan) 

d.  The  plans  for  the  six  areas  (as  reported  in  the  August  1985  Quarterly  Briefing) 

The  panel  found  the  September  19  plan  a  reasonable  beginning  toward  aiming  the  Program  in 
a  positive  future  direction.  In  general,  problems  are  reflected  in  disparities  among  die  general  goals 
and  strategies  expressed  in  the  plan  and  the  details  of  the  rest  of  the  plan,  the  current  area  activities, 
and  the  current  Program  organization. 

These  problems  are  explored  in  this  report.  A  brief  history  of  the  STARS  Program  is 
presented  in  the  next  Section  to  provide  a  context  for  understanding  the  panel's  conclusions.  In 
Section  3,  an  overview  of  the  panel's  major  findings  is  presented,  followed  by  a  detailed  discussion 
of  these  findings  in  Section  4.  The  panel's  recommendations  are  recapped  in  Section  5,  and  a  brief 
synopsis  of  the  panel's  assessment  appears  in  Section  6. 

2  0  HISTORY  OF  STARS 

The  impetus  for  the  STARS  Program  can  be  traced  back  to  an  April  1979  statement  to 
Congress  by  Ruth  Davis,  then  the  Deputy  Under  Secretary  of  Defense  for  Research  and  Advanced 
Technology.  She  announced  that  the  Department  of  Defense  (DoD)  would  launch  a  software 
technology  initiative.  Planning  for  the  Program  itself  began  in  FY80-81  when  the  Office  of  the 
Secretary  of  Defense  (OSD)  used  their  study  fund  to  plan  for  a  DoD  "Software  Initiative."  A 
workshop  was  held  in  Raleigh,  North  Carolina  in  FY83  during  which  the  original  proposal  for  the 
STARS  Program  was  presented.  The  general  goals  for  the  program  were  announced  to  be  to 
improve  the  productivity,  quality,  and  predictability  of  DoD  software. 

Originally,  the  Program  comprised  nine  areas:  Human  Resources,  Acquisition,  Human 
Engineering,  Application-Specific,  Project  Management,  Support  Systems,  Systems,  Measurement, 
and  Technology  Transition  (the  Software  Engineering  Institute). 

In  FY83  STARS  received  $2  million  funding  from  the  Defense  Advanced  Research  Projects 
Agency  (DARPA).  Congress  limited  FY84  funding  to  $1  million  because  it  felt  that  STARS  was  not 
adequately  justified  as  a  program  ([6]  summarizes  the  arguments  of  need  and  rationale  for  the 
STARS  program).  An  additional  $4  million  was  reprogrammed  from  the  Army  and  a  plan  was 
developed  to  disperse  the  funds.  This  plan  was  published  in  March  of  1984  and  a  task  force  was 
formed  to  elaborate  the  plan  and  begin  its  execution. 

At  this  time  the  nine  STARS  areas  were  reduced  to  six.  Project  Management  and  Acquisition 
were  combined.  Technology  Transition  was  seen  as  deserving  of  special  attention  under  a  separate 
program.  Human  Engineering  was  distributed  into  the  Human  Resources  and  Support  Systems 
Areas.  Support  Systems  was  divided  into  a  Software  Engineering  Environment  Section  and  a 
Methodologies  Section  that  incorporated  the  Systems  Area  and  the  Ada  Joint  Program  Office  (AJPO) 
Methodologies  Project.  Teams,  composed  of  volunteers,  were  formed  to  plan  accomplishments  for 
each  of  the  six  resulting  technical  areas. 

In  FY84  a  third  new  STARS  Director  was  appointed.  The  STARS  Director  and  the  service 
managers  from  the  Army,  Navy,  and  Air  Force  attended  a  retreat  in  July  1984,  and  there  shifted  the 
STARS  Program  emphasis  to  the  implementation  of  a  Software  Engineering  Environment  (SEE). 
The  technical  area  teams  were  told  to  produce  a  six -year  plan  and  fit  their  projects  into  the  context  of 


the  new  STARS  emphasis.  The  upper  management  of  the  program  was  now  operating  by  committee 
and  consensus. 

Up  to  this  time,  committee  members  had  been  mainly  volunteers,  paid  by  their  home 
organizations.  Beginning  in  FY85  their  salaries  were  paid  by  STARS,  but  they  were  still  employees 
of  their  various  agencies.  To  date,  Area  committee  members  are  federal  employees. 

Because  of  continuing  management  problems,  the  Program  Director  resigned  and  was 
replaced.  The  new  acting  director  started  just  in  time  to  attend  a  STARS  meeting  in  Virginia  Beach, 
Virginia,  in  February  1985,  where  some  progress  was  made  on  developing  the  program.  Although 
the  Services  voiced  strong  support  of  STARS  at  a  subsequent  1  May  1985  meeting  in  San  Diego, 
California,  program  management  problems  continued  and  increased  into  September  of  1985  when 
the  priority  of  the  issue  was  elevated  to  the  level  of  the  Under  Secretary  of  Defense.  The  Director, 
Computer  Software  and  Systems,  asked  the  SJPO  to  have  an  assessment  of  the  value  and  future  of 
the  program  performed.  The  basis  was  to  be  the  September  19, 1985  STARS  plan. 

3.0  SYNOPSIS  OF  THE  PANEL'S  FINDINGS 

The  panel's  assessment  of  the  STARS  Program  is  outlined  in  this  Section  and  discussed  in 
greater  detail  in  Section  4.0.  While  the  panel  found  the  major  problems  as  discussed  in  these  two 
Sections,  it  should  be  emphasized  that  the  panel  also  felt  that  the  current  plan  and  the  current  state  of 
the  program  are  adequate  starting  points  for  arriving  at  a  desirable  and  beneficial  future  for  the 
STARS  Program. 

First,  the  panel  found  organizational  and  management  problems  related  to  accountability, 
consensus,  spending  authority,  and  contracting.  The  panel  feels  that  the  Program  cannot  succeed 
with  the  current  committee  management  approach.  Although  OSD,  Computer  Software  Systems 
Directorate  (CSSD),  and  SJPO  management  have  consistently  opposed  a  committee  approach,  they 
have  not  been  successful  in  avoiding  the  diffusion  of  authority  now  found  in  the  Program.  The 
panel  recommends  a  strong  centralization  of  authority  and  accountability  with  associated  spending 
and  contracting  capabilities. 

Second,  the  panel  found  that  the  STARS  Program  has  in  the  past  consistently  lacked  an 
appropriate,  unifying  theme.  The  panel  feels  that  the  Program  cannot  reach  required  levels  of 
organization  without  such  a  theme.  Working  back  from  an  assessment  of  the  desired  post-STARS 
situation,  the  panel  developed  a  STARS  Program  theme  that  would  achieve  its  vision  of  the  future. 
In  short,  this  theme  and  vision  entails  the  fostering  and  assuring  of  the  continued  existence  of 
competitive  marketplaces  in  software  engineering  tools,  methods,  and  practices  and  in  Mission 
Critical  Computer  Resource  (MCCR)  software.  The  theme  is  a  modification  of  the  current  theme  in 
which  the  Program's  emphasis  is  shifted  away  from  building  environments  and  toward  stimulating 
the  marketplace  so  the  Government  and  its  contractors  can  buy  all  or  part  of  the  environments  they 
require. 

Third,  the  panel  assessed  each  Area's  goals,  plans  and  activities,  as  described  in  the  19 
September  Program  Plan  and  the  August  Quarterly  Briefings,  and  found  incompleteness, 
inconsistency  and  a  lack  of  coordination.  These  problems  stem  primarily  from  the  absense  of  strong 
top-down  direction  within  the  context  of  a  unifying  theme.  The  panel  used  its  suggested 
marketplace-based  theme  to  identify  several  possibilities  for  improvement  in  the  Areas'  plans.  Most 
of  these  improvements  fall  within  the  scope  of  the  SEE  Area  for  which  the  panel  suggests  a  major 
strategic  re-adjustment. 

Finally,  the  panel  found  major  problems  in  the  Program's  execution  and  again  used  its 
suggested  marketplace  theme  to  identify  several  actions  that  can  be  taken  to  spark  execution.  The 
panel  feels  that  those  existing  tasks  consistent  with  the  marketplace  theme  should  be  continued  or 


accelerated,  and  that  other  tasks  should  be  adjusted,  or  terminated.  In  addition,  the  panel  feels  that 
attention  must  be  rapidly  brought  to  bear  on  obtaining  intensive  industry  involvement  in  STARS 
activities. 

4.0  DETAILS  OF  THE  PANEL'S  FINDINGS 

This  section  presents  the  panel's  findings  with  respect  to  the  STARS  Program's 
management,  strategy,  and  technical  areas.  Included  with  the  findings  are  summaries  of  specific 
recommendations  for  improvement 

4.1  Management 

Management  problems  are  the  most  critical  ones  that  STARS  must  solve.  These  problems 
start  at  the  top  of  the  organization  with  an  OSD/Tri- Service  committee  arrangement  that  has  proven 
unable  to  provide  the  objective,  timely  decision  making,  the  consistency,  and  the  direction  needed  for 
development  and  execution  of  an  integrated  program.  Administrative  problems— particularly  delays 
in  transferring  funds— have  complicated  these  problems.  Uncertainties  regarding  appropriations 
have  added  to  these  delays  with  the  result  that  awarding  contracts  and  beginning  effective  execution 
have  been  significantly  delayed.  The  management  problems  therefore  span  the  full  range  of  the 
Program's  management. 

STARS  is  a  joint  Program  needing  extensive  coordination  and  cooperation,  but  sound, 
proven  management  principles  require  that  the  Director  be  given  the  authority  to  accomplish  the  job. 
A  permanent  Director  must  be  appointed  immediately  and  given  the  recognized  authority  to  institute 
top-down  planning  and  control  and  to  eliminate  management  only  by  consensus.  The  Director  needs 
the  authority  (and  "political  power")  to  set  directions  for,  organize,  manage  and  administer  the 
Program. 

The  sometimes  paralyzing  OSD/Service  conflicts  over  distribution  of  authority  must  be 
resolved.  This  will  require  attention  at  management  levels  above  CSSD  and  SJPO.  Once  these 
conflicts  are  resolved,  the  STARS  Director  can  institutionalize  the  arrangements  required  to  provide 
effective  Program  planning  and  execution.  Undoubtedly,  a  number  of  organizations  throughout  the 
Services  will  be  involved.  Formal  arrangements  according  to  each  Service's  practices  should  be 
established  to  ensure  proper  lines  of  responsibility. 

To  effectively  support  the  resulting  centralized  management,  administrative,  and  particularly 
financial,  processing  must  be  organized  and  swift  all  along  the  chain.  Effective  administrative 
processes  need  to  be  established  for  contracting,  tracking  disbursements,  preparing  plans,  reviewing 
plans,  and  reviewing  accomplishments.  While  the  higher-level  management  issues  are  more 
important,  these  administrative  processes  are  essential  to  an  effective  STARS  program. 

In  addition,  there  must  be  an  effective  line  management  organization.  This  implies  that  the 
execution  organization  should  be  different  from  the  organization  previously  used  for  planning.  It 
also  implies  the  appropriate,  facilitating,  official  arrangements  must  be  made  with  and  within  DoD 
Components.  A  line  organization  with  clear  execution  authority  and  accountability  is  critical  to  the 
Program’s  success.  Under  the  current  STARS  organization,  the  Director  has  no  line  authority  over 
Service  Program  Managers,  their  Deputies,  or  Area  Coordinating  Team  (ACT)  leaders  (see  Figure 
1).  A  line  management  organization  with  clearly  defined  managerial  and  technical  responsibilities 
must  be  established.  Individuals  should  have  clear  objectives  for  which  they  are  held  accountable. 
The  organization  should  reflect  a  well-structured  program  with  a  hierarchical  division  of 
responsibility. 


With  respect  to  the  role  of  the  Technical  Area  Teams  within  a  line  organization,  the  panel 
recommends  that  these  become  adjuncts  to  the  individuals  (and  project  offices)  managing  the  Areas. 
Their  role  should  be  advisory  and  technical.  Steps  should  be  taken  to  ensure  their  competence  in  this 
technical  role.  Over  time,  the  emphasis  in  some  areas  will  shift  from  development  to  insertion.  As 
this  happens,  the  membership  of  the  teams  may  also  need  to  shift  to  ensure  the  best  mix  of  expertise. 

Management  attention  will  be  a  scarce  resource,  and  STARS  management  needs  to  define  the 
critical  success  factors  and  critical-path  activities  where  it  should  concentrate  its  attention.  To  do 
this,  the  STARS  strategy  must  be  clearly  defined.  This  strategy  should  then  be  provided  as  guidance 
to  other  levels  in  the  Program.  One  of  the  most  critical  factors  is  technical  quality,  and  the  Director 
should  take  steps  to  provide  an  appropriate  review  mechanism  to  help  ensure  this  quality. 

Finally,  the  Director  must  move  rapidly  to  establish  effective  lines  of  communication  and 
synergy  within  the  Program  and  between  die  STARS  Program  and  other  programs  and  activities. 
Effective  communication  and  coordination  mechanisms  must  be  established  among  the  Area 
Managers  and  Teams  to  improve  integration  and  exploit  success.  Technology  transfer,  marketing 
and  execution  relationships  with  other  Programs  and  activities  must  be  institutionalized  to  establish  a 
customer  base  for  the  STARS  Program,  forestall  duplication,  and  establish  an  accurate  "public" 
perception  of  the  Program. 

The  panel  identified  three  alternatives  for  restructuring  STARS  management  to  assist  in 
accomplishing  some  of  these  changes  in  program  management  and  administration.  Management 
Structure  Alternative  One  is  close  to  the  current  structure  and  therefore  would  result  in  little 
perturbation  of  the  existing  structure  (see  Figure  2).  However,  because  the  bubbles  at  the  bottom 
are  individuals  rather  than  committees  or  teams,  this  alternative  has  the  problem  of  inverting  rank 
among  Deputies  and  Service  Program  Managers  and  the  panel  thought  it  could  be  improved. 
Management  Structure  Alternative  Two  increases  the  SJPO  to  a  dimension  more  suitable  to  the 
Program's  size  (see  Figure  3).  It  also  divides  the  Program  into  two  segments,  Technology 
Development  and  Use,  introducing  the  latter  to  explicitly  address  the  insertion  of  technology. 
Management  Structure  Alternative  Three  is  a  refinement  of  Alternative  Two  that  was  suggested  as  a 
result  of  one  of  the  panel's  briefings  to  its  reviewers  (see  Figure  4). 

These  alternatives  are  offered  in  a  positive  attempt  to  make  more  concrete  the  relatively 
abstract  recommendation  to  form  a  line  organization  for  the  STARS  program.  Hopefully,  the  real 
organization,  when  formed,  will  be  even  better  and  reflect  the  many  concerns  that  need  to  be 
addressed--without,  of  course,  abandoning  the  principle  of  having  a  truly  accountable  line 
organization. 

4.2  The  Desired  Situation  After  STARS 

Understanding  the  panel's  observations  and  recommendations  regarding  the  Program's 
strategy  and  technical  areas  requires  an  understanding  of  the  panel's  "vision"  for  the  future.  The 
panel's  view  of  the  future  emphasizes  the  marketplaces  for  DoD-related  software.  It  articulates  the 
required  characteristics  of  the  post- STARS  situation  and  the  results  of  STARS  in  terms  of  the 
marketplaces  for  software  tools  and  methods  as  well  as  end-use  MCCR  software.  For  the  most  part, 
this  vision  is  related  to  mission,  economic,  organizational,  and  acquisition  concerns  rather  than 
technical  concerns.  The  panel  felt  that  all  of  these  aspects  are  essential  to  the  success  of  the  STARS 
Program. 

The  ultimate  goal  must  be  to  contribute  to  ensuring  the  ability  to  build  and  support  the  DoD's 
mission  critical  systems  of  the  early  1990's.  While  doing  this,  the  basis  must  be  established  for 
meeting  the  requirements  in  the  years  beyond.  STARS  is  intended  to  be  a  program  of  finite  length 
ending  with  the  institutionalization  of  the  improvements  it  has  fostered  and  a  suggested  process  foi 
further  improvement. 
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Mission  critical  systems  requirements  imply  MCCR  software  requirements  that  in  turn  imply 
requirements  for  the  future  software  state-of-practice  within  the  DoD  community.  To  achieve  this 
required  state-of-practice,  the  necessary  technology  must  exist  and  it  must  be  successfully  used 
throughout  the  DoD  community.  Extensive,  successful  use  will  require  sufficient  numbers  of 
competent  persons  in  the  community  plus  effective  support  for  comparative  evaluation  of 
alternatives,  successful  and  speedy  introduction  of  new  technology,  and  propagation  of  the 
technology  throughout  the  community. 

With  these  goals  in  mind,  the  panel  identified  fifteen  requirements  for  the  situation  after 
STARS  (see  Figure  5).  The  most  important  requirements  are  intelligently  establishing  software 
requirements  and  then  efficiently  moving  to  meet  these  requirements.  The  exact  degree  of  necessity 
or  desirability  of  the  other  thirteen  is  arguable,  but  the  panel  felt  they  were  all  at  least  strongly 
desirable. 

STARS  needs  a  vision  of  how  the  defense  community  should  operate  in  the  mid- 1990's  to 
develop  and  support  MCCR  software  within  the  context  of  these  requirements.  The  panel  believed 
that  this  vision  should  be  along  the  lines  of  that  in  the  rest  of  Section  4.2. 

The  defense  community  will  be  well  ahead  of  potential  adversaries,  improving  rapidly  to  stay 
ahead,  and  maybe  even  widening  the  gap.  Defense  software  will  be  meeting  its  requirements  with 
the  needed  quality,  on  time,  for  reasonable  cost  -  and  doing  so  predictably.  To  do  this,  technology 
will  be  flowing  rapidly  into  use  throughout  the  DoD  community  regardless  of  its  commerical, 
academic,  or  Government  origin.  Facilitating  this  will  be  a  regulatory  context  that  fosters  the  needed 
investment  by  providing  an  equitable  return  for  all  (including  the  Government)  and  a  compatibility- 
based  marketplace  for  the  technology,  particularly  as  it  is  embodied  in  software  tools  and  in  reusable 
MCCR  software. 

Key  to  the  functioning  of  the  tools'  marketplace  will  be  standardized  interfaces  among 
environments  that  allow  transfer  of  (at  least)  work  products,  and  (at  most)  all  types  of  software  tools 
and  users.  These  standardized  interfaces  will  be  accompanied  by  acquisition  guidelines,  certification 
tests,  and  processes  for  development.  The  resulting  high  degree  of  compatibility  will  provide  a  basis 
for  the  marketplace's  success  and  the  cumulative  improvement  of  the  whole  community  while 
allowing  competitive  advantage  to  be  obtained  by  true  innovation  rather  than  by  "re-inventing  the 
wheel."  Among  other  things,  the  compatibility  will  ease  the  transfer  of  software  to  logistics 
organizations  and  potentially  provide  significant  cost  avoidance  because  of  the  reuse  of  software 
across  DoD  programs. 

Each  DoD  contractor  will  be  able  to  exercise  a  range  of  options  from  having  his  own  unique 
environment,  with  only  the  ability  to  exchange  work  products  with  other  environments,  to  having  a 
completely  compatible  environment.  Those  contractors  using  environments  that  are  compatible  with 
the  standard  interfaces  beyond  the  work  product  level  will  have  the  significant  advantage  of  being 
able  to  acquire  (and  also  to  lease  and  sell)  conforming  tools  in  the  marketplace.  This  variety  of 
situations  will  allow  a  range  of  Government  Furnished  Equipment  (GFE)  rather  than  contractor- 
furnished  approaches  to  Government  acquisition  of  end-application  software. 

DoD  will  be  an  intelligent  buyer  of  software  and  an  investor  in  software  technology.  The 
marketplace  will  be  broad  and  the  Government  will  be  able  to  obtain  the  best  software  and  software 
technology,  perhaps  obtaining  different  pieces  or  services  from  different  vendors.  Every  project  will 
be  able  to  rapidly  establish  an  appropriate  environment,  or  rapidly  modify  its  existing  environment, 
to  meet  the  needs  of  and  constraints  upon  the  project.  It  will  also  be  possible  to  rapidly  produce 
initial  and  subsequent  versions  of  MCCR  software.  Productivity  and  quality  levels  will  be  greatly 
improved  over  the  present  MCCR  software  situation. 
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Figure  5.  Interfaces  for  STARS  "Standardization1 


4.3  Program  Strategy 

The  Program's  general  strategy,  as  presented  in  the  19  September  Program  plan  emphasizes 
the  development  and  propagation  of  software  technology.  The  strategy  reflects  a  "Software 
Environments”  theme  under  which  all  activities  focus  on  developing  improved  environments  and 
their  more  wide-spread  use  [7].  As  a  result,  the  Program’s  activities  and  organization  emphasize 
technology  research  and  development. 

The  vision  of  a  future  world  presented  in  the  previous  section  suggests  a  marketplace  theme 
for  the  STARS  program.  This  theme  has,  as  its  central  characteristic,  the  Government-led 
development  of  competitive  marketplaces  for  both  MCCR  software  and  the  methods  and  tools  for 
creating  and  evolving  this  software.  Under  this  theme,  attention  must  still  be  directed  to  improving 
software  technology  and  to  the  extent  of  its  use.  Still,  the  marketplace  theme  shifts  attention  to 
exploitation  of  marketplace  mechanisms  to  achieve  initial  improvement  and  provide  for  continued 
improvement. 

It  is  the  panel's  judgement  that  a  marketplace  theme  is  considerably  better  for  meeting  the 
STARS  Program's  goals.  This  conclusion  was  reached  after  considering  the  implications  of  the 
requirements  for  the  future  world  discussed  previously,  considering  various  alternatives  for  creating 
this  world,  and  factoring  in  the  panel  s  understanding  of  the  industrial  and  political  situation  in  the 

U.S. 


The  following  section  analyzes  the  Program's  current  general  strategy.  Then  it  discusses  the 
suggested  marketplace  theme:  what  it  is;  why  it  is  valuable;  and  how  it  can  provide  an 
implementation  strategy  for  the  STARS  program.  Finally,  recommendations  are  made  for  several 
execution-related  actions  that  must  be  taken  to  turn  the  program  in  the  right  direction. 

4.3.1  Current  Program  Strategy 

The  general  strategy  reflected  in  the  19  September  program  plan  has  three  key  aspects.  First, 
management  is  centralized  within  OSD.  Second,  there  is  a  focus  on  several  key  technology 
improvement/ insertion  issues:  automation  of  software  creation  and  evolution,  reuse  of  software 
components,  use  of  advanced  design  technology,  standardization,  acceleration  of  technology 
transition,  improvement  of  DoD  business  practices,  and  the  creation  of  a  large  and  capable  work 
force.  Finally,  the  Program  is  organized  along  traditional  technology  development  subject  areas. 

The  major  problem  is  that  this  strategy  does  not  address  establishing  a  basis  for  continued 
improvement— it  focuses  on  an  initial  (and  significant)  improvement  in  capability  without  direct 
attention  to  further  improvement.  By  focusing  on  the  end  to  be  achieved  rather  than  the  means  for 
achievement,  the  strategy  directs  the  Program’s  attention  to  technology  research  and  development 
without  capturing  and  preserving  the  process  by  which  improvement  can  be  obtained. 

4.3.2  Proposed  Marketplace  Strategy 

Based  on  the  requirements  for  the  panel’s  envisioned  future  world,  its  understanding  of  the 
lndustnal/political  situation  surrounding  the  STARS  program,  and  a  consideration  of  various 
alternatives,  the  panel  concluded  it  would  be  best  to  base  the  Program's  strategy  upon  the 
exploitation  of  marketplace  mechanisms.  The  panel  felt  that  such  a  strategy  would  lead  to  the  needed 
near-term  improvement  being  sought  as  a  direct  effect  of  the  STARS  program  and  also  provide  the 
basis  for  continued  improvement  being  sought  as  an  indirect  effect. 

The  marketplace  theme  causes  STARS  to  focus  on  fostering  a  competitive  supplier 
community  for  the  tools,  processes  and  end-application  components  needed  to  produce  and  maintain 
software  meeting  DoD  requirements.  The  intent  is  to  foster  marketplaces  in  the  underlying 


technologies  needed  to  meet  the  requirements  for  DoD  software  in  the  near-term  and  the  future. 
While  active  DoD  involvement  and  some  DoD  investment  will  always  be  required,  STARS  should 
think  in  terms  of  establishing  a  situation  where  DoD  state-of-the-practice  software  technology  and 
end-application  software  needs  are  naturally,  quickly  and  cost  effectively  met  via  free -enterprise 
marketplaces. 

Two  related  marketplaces  must  be  created,  each  presenting  distinct  inherent  requirements. 
Establishing  a  marketplace  for  tools  and  processes  requires  achieving  high  degrees  of  compatibility, 
significant  private  sector  investment,  government  acquisition  policies/practices  that  contribute  rather 
than  inhibit,  and  the  active  use  of  comparative  evaluation  mechanisms.  Establishing  marketplaces  for 
end-application  components  involves  creating  a  number  of  marketplaces  for  the  different  applications 
areas— markets  with  mainly  one  buyer,  E>oD.  The  solutions  to  problems  such  as  rights  of  data  may 
differ  from  solutions  to  the  same  problems  in  the  tools  marketplace  because  of  DoD's  role  as 
dominant  consumer  of  MCCR  products. 

Several  characteristics  are  desirable  for  both  types  of  marketplaces.  The  products  available  in 
the  marketplaces  should  cover  the  full  software  system  life  span.  The  products  should  not  just  stress 
and  support  the  automated  aspects  of  software  production  and  creation  but  they  should  also  address 
other  aspects;  courses  intended  to  improve  human  skills,  for  example,  would  be  a  reasonable 
marketplace  product.  Finally,  the  marketplaces  should  admit  a  variety  of  options  with  respect  to 
GFE  policy  towards  products. 

Several  other  national  software  technology  programs  (e.g.,  ESPIRIT,  Japanese  Fifth 
Generation)  have  recognized  the  value  of  a  marketplace-based  strategy  that  emphasizes  industry 
involvement  in  both  producing  an  initial  improvement  in  software  technology  and  participating  in  a 
free-enterprise  marketplace  to  provide  continued  improvement.  The  benefits  of  using  this  strategy 
for  the  United  States'  program  include;  (1)  attraction  of  private-sector  investment,  (2)  attraction  of 
expertise  outside  of  that  obtained  by  direct  contract,  assurance  of  product  support,  and  (3) 
development  of  a  partnership  rather  than  adversary  relationship  with  industry. 

In  the  remainder  of  this  Section  we  discuss  the  Marketplace  Theme,  its  perceived  value  and 
ideas  for  its  creations  in  greater  detail.  We  also  demonstrate  how  the  Marketplace  Theme  can  provide 
a  strategy  for  STARS  and  how  it  can  be  implemented  and  executed  in  the  program. 

4.3.2. 1  Marketplace  in  Software  Engineering  Tools  and  Methods 

This  marketplace  concerns  the  technology  used  to  create  and  evolve  software.  The  products 
within  this  marketplace  pertain  to  the  tools  and  methods  that  software  practitioners  (both  in  and 
outside  government)  use  to  produce  and  maintain  MCCR  software  systems.  Creating  this 
marketplace  requires  solving  several,  primarily  technical,  problems.  Chief  among  them  is  providing 
for  technical  compatibility  through  standardized  interfaces.  Attracting  investment  from  the  private 
sector,  instituting  appropriate  acquisition  policies/practices,  and  providing  a  comparative  evaluation 
cabability  are  the  other  key  issues. 

4.3.2. 1.1  Technical  Compatibility 

There  are  a  variety  of  different  ways  in  which  we  can  think  about  compatibility  among 
software  engineering  environments  .  Five  major  ones  are: 

( 1 )  Lower  level  data  exchange.  This  encompasses  basic  data  communication  mechanism 
and  media  (e.g.,  magnetic  tape)  compatibility  between  collections  of  tools.  It  can  relate 
to  anything  from  mail  and  file  transfer  to  the  transfer  of  database  contents,  possibly  even 
based  on  a  fairly  high-level  model  such  as  the  entity-relationship  model. 


(2)  Work  product  exchange.  This  refers  to  the  degree  to  which  a  full  set  of  software  related 
work  products,  in  machine  readable,  manipulable,  and  analyzable  form,  can  be 
transferred  between  tool  collections.  This  potentially  includes  all  the  deliverables  called 
for  by  DoD-STD-2167  and  other  standards,  plus  required-but-not-deliverable  products 
and  other  work  products. 

(3)  Tool  transportability.  This  can  concern  a  variety  of  issues  from  moving  self-contained 
tool  sets  to  moving  an  individual  tool  that  works  effectively  with  many  other  common 
tools,  services,  and  data.  Tool  transportability  requires  the  definition  of  boJi 
information  and  invocation  interfaces.  Among  these  interfaces  are  those  pertaining  to  the 
operating  system;  the  data  management  subsystem;  the  human  interface  management 
subsystem,  and  to  generic  tools  and  tool  fragments. 

(4)  Functional  scope.  Two  tool  collections  may  differ  in  the  functionality  they  provide. 

They  may  overlap  completely  or  there  may  be  relatively  little  or  no  overlap.  Functional 
scope  compatibility  is  important  when  considering  whether  two  environments  can  be 
combined  or  used  in  tandem. 

(5)  Transparency  to  users.  This  concerns  the  degree  to  which  differences  in  tool 
implementation  or  installation  are  hidden  from  the  user.  This  can  be  important  not  only 
for  users  to  be  able  to  easily  move  from  project  to  project  but  alsowhen  considering  the 
compatibility  of  the  user  aids  (e.g.,  documentation,  manuals,  etc.)  transferred  along  with 
work  products  or  tools. 

While  couched  in  terms  of  tool  compatibility,  it  should  be  understood  that  the  remarks  apply  equally 
to  the  compatibility  of  the  methods  supported  by  the  tools.  For  example,  methods  can  be  more  or 
less  compatible  with  regard  to  user  transparency,  depending  on  the  degree  to  which  insignificant 
definitional  details  (perhaps  only  pertinent  to  implementing  the  supporting  tools)  are  hidden  from  the 
user. 

The  most  obvious  interfaces  are  external,  information-oriented  ones;  obvious  in  that  they 
pertain  to  the  information  flow  between  an  environment  and  the  world  outside  the  environment. 
Interfaces  will  also  be  needed  for  information  structures  internal  to  an  environment,  including: 
interfaces  for  lower  level  work  products  internal  to  the  environment  (e.g.,  an  annotated  data  flow 
graph);  prominent  inter-tool  interfaces  that  may  not  be  work  products  (e.g.,  DIANA);  specifications 
used  as  input  to  tool  generators;  specifications  of  scripts  or  procedures;  and  specifications  of 
guidance  and  assistance  information  such  as  used  in  on-line  tutorials  and  help  documents. 

Information  interfaces  pertain  to  the  structure  and  content  of  the  information  flowing  across  a 
boundary.  Invocation  interfaces  are  needed  as  well;  these  interfaces  specify  the  ways  in  which 
control  can  be  passed  back  and  forth  across  the  boundary  over  time.  There  are  a  variety  of 
candidates  for  invocation  interfaces  in  a  typical  environment;  among  them  are  the  interfaces  to  the 
database  subsystem,  the  human  interface  subsystem,  the  operating  system,  the  communications 
subsystem,  and  generic  tools  or  tool  fragments  (e.g.,  invocation  interfaces  to  editors  and  compilers). 
These  and  other  invocation  interfaces  are  discussed  in  the  Integration  and  Compatibility  Framework 
Section  of  the  Operational  Concept  Document  [4],  The  Common  APSE  Interface  Set  (CAIS)  effort 
will  continue  to  address  the  operating  system  invocation  interface,  but  other  efforts  (or  an  expanded 
CAIS  effort)  are  needed  to  adequately  address  all  of  the  issues  involved. 

Ensuring  the  desired  high  levels  of  compatibility  requires  not  only  rigorous  interface 
specification,  but  also  criteria,  metrics,  and  evaluation  and  certification  capabilities.  In  addition,  the 
interface  definition  process  must  capitalize  on  future  technological  improvements  as  well  as  provide 
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for  extensibility.  Finally,  some  degree  of  standardization  may  be  required  (at  least  at  the  defacto 
level),  along  with  other  actions  by  the  Government,  in  order  to  bring  the  desired  levels  of 
compatibility  into  existence  and  mature  them  over  time. 

Figure  6  shows  several  different  types  of  standards  implied  by  the  differing  types  of 
compatibility  discussed  above.  Lower  level  data  exchange  directly  implies  the  need  for  at  least  one 
path  (e.g.  magnetic  tape,  data  communications)  between  environments  meeting  the  time,  volume, 
and  security  requirements  for  a  given  exchange.  Work  product  exchange  implies  the  need  for  a 
framework  for  the  elements  being  transferred  and  specifications  for  the  individual  informatioh 
fragment  types  and  their  interrelationships.  Tool  transportability  implies  standardization  of  the 
formats  for  information  in  the  database  or  passed  between  tools  and  the  invocation  interfaces  used  by 
the  tools.  Less  directly,  tool  transportability  depends  on  consistency  in  the  delivery  of  the 
functionality  required  by  the  tools  and  consistency  among  the  human  interface,  documentation  and 
help  facilities.  Consistency  in  functional  scope,  if  desired,  could  be  enforced  using  a  standard 
acceptability  benchmark.  User  transparency  implies  at  least  similar  human  interfaces  and  available 
functionality  Also  of  interest,  both  for  achieving  consistency  in  general  and  for  facilitating  the  use 
of  tool  generators,  are  the  notations  and  processes  available  for  specifying  interfaces. 

4.3.2. 1.2  Other  Key  Issues 

In  addition  to  interfaces  supporting  technical  compatibility,  establishing  a  marketplace  in 
software  engineering  tools  and  methods  requires:  (1)  the  attraction  of  private-sector  investment,  (2) 
the  development  of  appropriate  acquisition  policies  and  practices,  and  (3)  the  provision  of  a 
comparative  evaluation  capability.  The  Business  Practices  Area  must  focus  its  attention  on  these 
issues. 


Private  sector  investment.  It  is  unreasonable  to  expect  that  taxpayer  money  will  be  used  to 
underwrite  all  the  improvement  that  will  be  needed;  investment  from  the  private  sector  must  be 
actively  pursued.  Small  investors  and  innovative  firms  can  be  attracted  if  they  can  enter  the 
marketplace  with  a  low-investment  product  that  will  work  compatibly  with  other  products  found  in 
the  marketplace.  In  addition,  the  larger  the  marketplace,  the  more  attractive  it  will  be  to  investors  and 
firms  at  all  levels.  Technical  compatibility  beyond  just  the  DoD  community  can  thus  be  beneficial  to 
DoD 


While  competition  among  suppliers  is  healthy,  DoD  will  benefit  most  if  investors  build  on 
each  other,  rather  than  duplicate  each  other.  The  majority  of  products  in  the  marketplace  should 
therefore  be  small-scale  ones  that  can  be  used  in  a  variety  of  situations  on  a  mix-and-match  basis.  In 
addition,  the  practice  of  adding  value  to  existing  products  should  be  routine  and  extensive. 

Uncertainty  will  scare  off  private  sector  investment.  DoD  can  help  alleviate  this  uncertainty 
b\  doing  the  research  and  development  to  demonstrate  feasibility.  It  can  also  help  by  removing  the 
uncertainty  from  its  own  future  marketplace-related  actions  and  policies.  (For  example,  an 
announced"  commitment  to  buy  in  the  future  in  a  certain  fashion  can  be  a  powerful  motivator  for 
potential  suppliers. )  The  Government  needs  to  form  a  pattern  of  procurement  requests  for  products 
using  well-defined  criteria,  and  industry  needs  to  feel  that  the  definition  is  stable  enough  so  as  not  to 
void  industry  investment.  The  Government  should  then: 

( 1 )  Insist  on  compatibility  in  all  related  procurements  (even  hardware). 

(2)  Use  financial  and  other  incentives  to  speed  compliance:  use  metrics  to  better  ensure 

results  and  compliance. 

(3)  Emphasize  compatibility  and  currency  of  technology  in  RFP  evaluation  criteria. 
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Acquisition  policies  and  practices.  As  indicated  the  Government's  acquisition  policies  and 
practices  should  remove  uncertainty  from  the  marketplace  as  much  as  possible.  In  addition,  these 
policies  and  practices  should  serve  to  create  demand  in  the  marketplace,  lead  to  the  Government 
obtaining  better  MCCR  software,  accelerate  technology  insertion,  and  accomo-date  a  range  of  GFE 
policies.  Creating  improved  acquisition  policies  and  practices  will  require: 

(1)  Definition  of  an  underlying,  uniform  interface  model  supporting  independent  tool 
development  and  integration. 

(2)  Definition  of  the  commitments  required  of  a  vendor  (support  levels,  documentation, 
performance  standards,  etc.). 

(3)  Regular  identification  of  a  "shopping  list"  of  automated  tools  and  methods  (based,  in 
part,  on  assessments  of  valuable  future  technology). 

(4)  Provision  of  an  evaluation  and  certification  service  as  well  as  vehicles  forintegrating  the 
tools  and  methods  into  the  STARS  environments  and  the  marketplace. 

Also,  issues  related  to  rights  in  data,  software  licensing  and  mixed  ownership  must  be  studied  with 
respect  to  their  effects  on  the  marketplace. 

Comparative  evaluation  capability.  Tools  and  methods  products  will  be  successful  in  the 
marketplace  only  if  they  are  of  production  quality  and  are  fully  supported.  The  consistent  use  of 
metrics  across  the  community  will  allow  developers  and  the  entire  market  to  readily  judge  new 
products  and  interpret  reported  experience  with  older  ones.  This  will  require  an  evaluation  capability 
that  relies  on  measurable  characteristics,  helps  in  certifying  new  marketplace  entries,  and  assists  in 
comparing  among  the  various  alternatives  that  will  emerge.  For  the  sake  of  time,  the  DoD  may  need 
to  supply  its  own  evaluation  capability. 

4. 3. 2. 2  MCCR  End-Application  Component  Marketplace 

The  products  appearing  in  this  marketplace  are  components  of  MCCR  software  systems. 
The  technical  problems  discussed  for  the  software  engineering  tools  and  methods  marketplace  are 
pertinent  here.  Many  of  the  mechanisms  and  changes  developed  for  the  Tools  and  Methods 
Marketplace  will  be  useful  in  the  Component  or  Generator  marketplaces.  Therefore,  these 
mechanisms  and  changes  must  be  prepared  with  a  concern  for  the  MCCR  Marketplace  as  well.  Each 
application  marketplace  can  be  expected  to  have  fewer  suppliers  and  only  a  few  buyers. 

In  addition,  a  number  of  non-technical  issues  make  the  development  of  this  marketplace 
complex,  especially  as  it  relates  to  reusable  software.  These  issues  include:  classified  software; 
proprietary  interests;  rights  in  data  (e.g.,  derivative  works);  undesirable  foreign  technology  transfer; 
libraries;  warehousing;  cataloguing  and  retrieval;  support;  incentives;  royalities;  and  making  software 
a  recurring  business.  Many  of  these  issues  have  received  attention,  but  much  more  effort  is  required 
before  they  are  solved  entirely.  The  Business  Practices  and  Applications  Areas  will  be  particularly 
critical  in  this  regard. 

4. 3. 2. 3  Desirable  Marketplace  Characteristics 

Both  types  of  marketplaces  must  exhibit  several  characteristics.  First,  the  products  must 
collectively  span  the  full  software  life  cycle,  covering  all  activities  from  initial  conception  to  final 
retirement.  And  the  products  must  not  merely  be  (closed-system)  implementations;  definitions  and 
designs  must  also  be  available. 
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Second,  the  marketplaces  must  exhibit  concern  for  the  people  who  will  use  the  technology 
This  means  that  the  products  must  be  accompanied  by  appropriate  documentation  and  that  training 
and  education  products  should  be  present  in  the  marketplaces. 

Third,  the  marketplaces  must  admit  a  wide  variety  of  options  with  respect  to  the  issue  of 
GFE.  Different  parts  of  DoD  have  (and  will  probably  continue  to  have)  different  policies  regarding 
provision  of  software  engineering  environments  and  tools.  Some  desire  to  supply  them  to  the 
contractor  and  some  prefer  to  give  the  contractor  complete  freedom  (and  responsibility)  to  use 
whatever  he  wants.  In  practice  and  in  theory  a  whole  spectrum  exists  between  a  strict  GFE 
requirement  and  total  freedom. 

The  panel  identified  several  pros  and  cons  of  this  spectrum's  end-points.  Arguments  for  GFE 
include  improved  maintenance  and  interoperability  while  opponents  of  GFE  argue  that  GFE  is 
costly,  contributes  to  receiving  low  quality  products,  and  presents  a  delay  in  the  insertion  of  new 
technology  into  the  community.  On  the  other  hand,  proponents  of  total  freedom,  supported  by 
competitive  marketplaces,  argue  that  this  leverages  captial,  naturally  selects  for  quality,  and 
incrementally  inserts  technology.  Issues  of  control,  interoperability,  and  compatibility  argue  against 
the  competitive  marketplace  end  of  the  spectrum.  The  panel  had  strong  beliefs  regarding  this  subject 
but  recognized  that  STARS  cannot  (and  should  not)  dictate  policy  on  this  issue. 

The  panel  observed  that  the  issue  of  GFE  is  really  totally  separate  from  the  issue  of  whether 
or  not  STARS  should  adopt  the  suggested  marketplace  theme.  DoD  could  GFE  products  regardless 
of  whether  they  come  from  the  marketplaces  or  are  developed  "in-house."  Thus,  any  decisions 
regarding  GFE  do  not  affect  (or  negate)  the  recommendations  presented  here  for  implementing  and 
executmg  the  STARS  Program  under  a  marketplace-based  strategy. 

4.3  2  4  Why  the  Marketplace  Theme  is  Important 

The  theme  of  creating  and  using  marketplaces  is  important  because  it  lays  the  foundation  for 
continued,  evolutionary  improvement  toward  the  envisoned  future  world  discussed  previously.  It 
helps  to  meet  the  following  requirements  for  this  future  world:  attracting  and  leveraging  private- 
sector  investment;  fostering  a  high  degree  of  portability  for  software  tools;  speeding  the  flow  of 
technology  into  wide-spread  use;  and  supporting  the  reuse  of  software  system  components. 

Several  other  national  software  technology  improvement  programs  have  emphasized 
govemment/industry  cooperation  for  their  programs.  The  different  cultures  surrounding  these  other 
programs  have  led  to  details  that  differ  from  those  suggested  here.  However,  the  sense  is  somewhat 
the  same  —  use  Government  influence  to  shape  and  prime  the  ways  that  software  and  software 
technology  are  acquired.  Within  the  context  of  the  American  system,  this  intent  rather  naturally  leads 
to  a  marketplace-based  strategy  that  exploits  the  capabilities  of  the  U.S.  industrial/political  system. 
The  marketplace  theme  is  important  in  that  it  exploits  an  existing  system  to  achieve  its  results. 

Finally,  the  marketplace  theme  is  important  because  it  leads  to  direct  and  extensive  industry 
involvement  in  STARS  activities.  Directly  acquired  products  of  the  STARS  program  will  be 
contracted  under  Federal  Acquisition  Regulations  (FAR).  FAR  part  27  of  the  DoD  FAR  Supplement 
covers  rights  in  technical  data  and  software,  and  is  very  explicit;  products  (particularly  software) 
developed  with  the  Government's  money  are  in  the  public  domain.  By  developing  products  under 
private-sector  investment,  and  letting  the  investors  retain  proprietary  rights,  industry  should  actively 
participate  in  achieving  the  Program's  goals  and  objectives,  with  numerous  benefits.  Needed 
additional  investment  will  be  generated  from  outside  STARS.  The  benefits  of  the  American  system 
(such  as  private  initiative,  competition,  and  the  desire  to  make  a  profit)  will  be  exploited.  Industry 
understands  the  necessity  to  supply  a  fully  supported  product  and  is  motivated  to  get  its  products 
used;  a  marketplace-based  strategy  will  therefore  foster  both  follow-on  support  for  products  and  a 


technology  push  from  within  industry.  Finally,  expertise  other  than  that  directly  contracted  for  will 
be  attracted  to  the  solution  of  DoD  problems. 

4.3. 2.5  Implementing  the  Marketplace  Theme 
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Creating  these  marketpla  <*s  requires  developing  a  customer  base,  a  base  of  private  funding, 
and  a  facilitating  regulatory  context.  To  do  this  through  the  STARS  program  requires  a  new 
implementation  strategy  that  emphasizes  market  force  exploitation  and  a  focus  on  buying  products 
rather  than  building  them  in-house.  This'  implementation  strategy,  in  turn,  requires  several 
immediate  shifts  in  die  execution  of  STARS  Program.  In  this  Section,  we  First  discuss  the  creation 
of  the  marketplace  in  general  and  then  propose  specific  implementation  and  execution  strategies  for 
STARS  that  foster  marketplace  creation.  Again,  this  theme  must  permeate  all  STARS  Areas,  but  the 
Business  Practices  Area  should  be  the  coordinating  area  for  these  issues. 

4.3. 2.5.1  Creating  the  Marketplaces 

The  primary  issues  in  marketplace  creation  are:  (1)  creating  customer  demand;  (2)  creating 
supplier  community  interest,  investment,  and  products;  and  (3)  creating  a  favorable  regulatory  and 
policy  context.  Secondary  issues  are  influencing  the  Program's  primary  customers  and  influencing 
the  supplier  community. 

The  Ada  effort  is  an  example  of  the  synergism  required  and  possible  in  regulatory, 
acquisition,  and  supplier  policy  to  support  the  desired  marketplaces.  Market  demand  was  created,  or 
reinforced,  by  the  letter  from  Dr.  DeLauer  that  established  an  Ada  policy  (even  though  the  letter 
basically  only  issued  a  revised  policy  draft  for  service  coordination,  and  did  not  legislate  that  policy 
officially).  Supplier  activity  started  because  of  a  void  in  programming  language  activities  in  DoD 
from  the  mid-1970's  to  early-  1980's,  and  the  perception  by  1980  that  the  Ada  language  was  going  to 
be  sufficiently  successful  for  the  DoD  to  demand  its  use.  Regulatory  synergism,  and  further  fuel  for 
supplier  interest,  came  in  the  form  of  the  Validation  Test  Suite  for  Ada.  This  device  both  assured 
venture  investors  that  there  was  a  way  to  measure  the  market  suitability  of  their  product,  and 
provided  customers  with  a  form  of  assurance  of  product  conformance.  The  original  intent  of  the  test 
suite  was  different,  namely,  to  control  the  proliferation  of  language  variants. 

Creating  customer  demand.  The  primary  customers  for  the  software  technology  resulting 
from  the  STARS  Program  are  DoD  Program  Managers.  Creating  customer  demand  therefore 
requires  that  the  Program  Managers  be  educated,  legislated,  and  politically  encouraged  to  require 
STARS-related  products  to  be  supplied,  used  and  supported  in  their  systems.  Program  Managers 
must  be  made  to  feel  that  they  cannot  do  without  the  products,  methods  and  practices  resulting  from 
the  STARS  Program.  Education  and  assistance  of  Program  Managers  must  then  be  a  very  high 
priority  item.  STARS  can  provide  the  assistance  by  demonstrating  benefits,  assuring  the 
development  of  needed  technology,  and  providing  money  for  expert  assistance,  tool  demonstrations, 
education,  etc. 

Creating  supplier  community  interest,  investment  and  products.  This  requires  a  perceived 
customer  demand,  a  favorable  regulatory  and  policy  environment  (to  be  discussed),  and  investor 
appeal.  Investor  appeal  is  needed  because  supplier  business  performance  is  measured,  by  the 
owners  of  the  supplier's  business,  in  terms  of  the  degree  of  return  on  investors’  capital  (spent  in 
creating  the  product  and  developing  its  market).  Various  approaches  to  assuring  a  return  on 
investment  should  be  investigated  and  the  more  promising  ones  should  be  actively  "marketed"  (along 
with  the  related  factors  discussed  above)  within  the  DoD  community  and  software  community  at 
large. 

Creating  a  favorable  regulatory  and  policy  context.  Critical  to  the  creation  of  a  favorable 
regulatory  and  policy  context  is  the  resolution  of  FAR  inconsistencies.  The  FAR  secretariat  is  in  the 
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process  of  re-issuing  both  civilian  agency  (and  NASA)  versions  of  the  FAR  and  the  DoD  FAR 
supplement.  These  two  regulations  differ  substantially  in  their  definition  of  software,  their 
delineation  of  data  rights  and  private  retention  of  software  rights,  and  in  the  protection  each  offers  to 
the  developer  (to  protect  supplier  interests)  and  to  the  Government  (to  protect  buyer  interests).  To 
achieve  standardization  and  reusability  in  the  software  area,  it  seems  necessary  to  consider  parallel 
revision  of  these  two  regulations  (as  far  as  they  pertain  to  software  products  which  are  likely  to  be 
common  to  DoD  and  non-DoD  agencies,  such  as  Ada-related  tools  and  environment  products). 

Secondary  regulatory  issues  concern  DoD-level  instructions  and  directives.  These  will 
probably  require  revision  as  determined  by  the  Business  Practices  Area,  and  as  determined  by  the 
Software  Engineering  Institute's  (SEI)  related  investigations.  Coordination  between  the  STARS  and 
SEI  business  practice  activities  is  mandatory  for  success  in  this  regard. 

Influencing  the  Program's  primary  customers.  In  order  to  influence  Program  Managers,  it  is 
necessary  to  understand  how  they  are  motivated.  Program  Manager’s  are  motivated  by  two  levels  of 
concerns:  ( 1 )  their  career  and  (2)  their  performance  with  respect  to  their  current  tasking  as  reflected 
by  fitness  reports.  The  first  concern  is  related  to  career  advancement,  considering  fitness  reporting, 
superior  satisfaction,  and  ability  to  move  in  a  long-term  direction.  This  can  be  in  conflict  with  near- 
term  tasking  (project)  performance,  particularly  where  project  performance  optimization  might 
require  decisions  that  are  politically  dangerous.  STARS  management  must  be  sensitive  to  the  issue 
of  risk  reduction  for  Program  Managers. 

Because  many  DoD  systems  have  several  Program  Managers  prior  to  operational  acceptance 
and  several  after  initial  fielding,  there  is  a  serious  conflict  between  a  single  Program  Manager's  goals 
of  delivering  within  cost  and  on  time,  and  the  long  term  program  life  cycle  costs  and  risks.  This 
second  level  of  concern  is  relatively  easy  to  cope  with,  as  long  as  STARS-related  needs  are  justified 
(proven)  to  the  Program  Manager  (and  reasonable  in  an  absolute  sense).  Marketing  STARS-related 
concepts  is  required.  Reinforcement,  following  decision  making ,  is  also  necessary  to  make  a  "sale" 
stay  "sold." 

Influencing  the  supplier  community.  This  issue  arises  because  STARS  has  a  significant 
budget  and  its  expenditures  can  significantly  influence  the  marketplace.  The  SEE  Area,  with  the 
largest  budget  portion  of  the  STARS  program,  has  the  most  possibility  of  influence.  Environment- 
related  expenditures,  properly  controlled  to  direct  supplier  attention  to  desired  topics,  should  be  a 
major  focus.  Significant  attention  should  be  given  to  desired  supplier  focus  with  regard  to  futuie 
environment-related  purchases.  This  includes  both  topical  areas  of  expenditure,  as  well  as 
requirements  levied  on  suppliers  (such  as  in  terms  of  standard  interfaces). 

If  the  19  September  Program  Plan's  strategy  is  followed,  an  additional  consideration  should 
be  the  breadth  of  marketplace  coverage  when  procuring  the  six  initial  designs  and  the  final  two 
substrates.  It  may  be  necessary  to  either  encourage  suppliers  to  have  a  wide  participation  of 
subcontractors,  or  otherwise  provide  a  larger  number  of  smaller  contracts.  Section  4.4.3  on  the  SEE 
Area  recommends  the  former.  This  is  necessary  to  expand  the  market  influence  of  SEE-related 
expenditures  beyond  two  main  SEE  suppliers. 

4. 3. 2. 5. 2  Implementation  Strategy 

The  19  September  Program  plan  essentially  proposes  to  do  "business  as  usual,'  that  is. 
develop  large-scale  products  using  existing  DoD  acquisition  approaches  In  this  section,  the 
problems  with  this  approach  are  first  identified  and  then  several  general  principles  to  guide  the 
development  of  a  detailed  implementation  strategy  are  discussed. 


The  STARS  Program  cannot  afford  to  conduct  its  business  as  usual.  It  is  well-recognized 
that  this  approach  is  synonomous  with  cost  overruns,  low  quality,  and,  in  some  cases,  entire 
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systems  never  being  delivered  at  all — STARS  must  expect  that  it  may  fall  prey  to  these  attendant 
problems.  STARS  must  try  out  new  ways  of  doing  business  or  it  will  become  another  large, 
ineffective  Government  program,  exhibiting  all  of  the  problems  it  is  supposed  to  be  helping  to  solve. 


A  disproportionate  share  of  STARS  resources  are  currently  targeted  toward  the  development 
of  various  technologies,  particularly  in  the  form  of  production-quality  tools  (a  ”build-it"  approach). 
Instead,  as  discussed  above,  STARS  should  be  stimulating  the  private  sector  to  provide  the  needed 
technology  including  tools,  methods,  and  reusable  application  components.  STARS  should  also  be 
helping  to  transform  the  DoD  into  an  intelligent  buyer  of  that  technology. 

One  can  look  to  the  Ada  Program  to  see  the  results  of  both  approaches.  The  build-it 
approach  has  been  employed  by  the  Army  and  Air  Force  environment  efforts  (the  ALS  and  AIE, 
respectively)  and  these  efforts  have  encountered  well-publicized  problems  in  the  area  of  costs, 
schedules,  and  performance.  The  market-stimulation  approach  has  resulted  in  a  number  of  efforts 
undertaken  by  private  industry  (e.g.,  ROLM/Data  General,  Verdix,  Digital)  that  have  met  with 
considerably  more  success. 

The  market-stimulation  approach  has  worked  for  the  Ada  Program  because  software  vendors 
have  had  a  reasonable  degree  of  assurance  that  a  market  for  Ada  products  exists.  At  the  same  time, 
the  DoD  has  successfully  maintained  control  over  the  language  through  the  compiler  validation 
process.  The  Ada  Program  can  provide  a  model  on  a  small  scale  of  what  needs  to  be  done  in  the 
DoD  on  a  much  larger  scale.  There  are  many  issues  that  are  not  understood  at  this  point.  Clearly, 
risk  is  involved  but  the  alternative  of  "business  as  usual”  is  no  alternative  at  all  for  a  program 
intended  to  solve  the  problems  caused  by  that  very  approach. 

Rather  than  being  in  the  business  of  developing  tools,  it  should  be  the  responsibility  of 
STARS  to  be  very  clear  about  what  it  needs,  that  is,  to  provide  clear  and  stable  specifications.  As 
much  as  is  possible,  these  specifications  should  be  expressed  in  an  objective  and  quantitative 
manner.  It  should  also  be  the  responsibility  of  the  STARS  Program  to  develop  mechanisms  for 
determining  adherence  to  any  given  set  of  specifications.  This  should  serve  as  a  model  for  DoD  as  a 
whole  in  procuring  at  least  major  portions  of  MCCR  systems  through  the  use  of  reusable  application 
components.  It  should  be  left  up  to  the  marketplace  to  be  innovative  in  producing  the  best  products 
according  to  these  criteria.  The  validity  and  reliability  of  the  valuation  and  selection  criteria  are 
obviously  critical  if  this  approach  is  to  succeed. 

The  role  of  each  of  the  areas  within  STARS  should  be  to  develop  rigorous  specifications 
reflecting  identified  DoD  requirements.  In  order  to  stimulate  meaningful  competition,  these 
specifications  and  associated  acceptance  criteria  must  be  made  public.  STARS  should  promote  the 
explicit  policy  that  the  DoD,  as  a  customer,  will  evaluate  competing  products  on  the  basis  of  these 
criteria. 


None  of  these  marketplace  stimulation-oriented  implementation  actions  are  simple;  but  all  of 
them  are  critically  necessary.  The  STARS  Program  will  have  to  evolve  a  detailed  Section.  The  panel 
feels  that  these  principles  have  general  utility  and  should  be  kept  in  mind  during  the  development  of 
any  strategy  for  STARS  implementation  regardless  of  whether  or  not  it  adheres  to  the  suggested 
marketplace  theme. 

Exploitation  of  marketplace  forces.  Since  direct  marketplace  creation  is  impossible,  the 
STARS  program  must  indirectly  shape  and  mature  the  marketplaces  through  its  activities.  First. 
STARS  must  estimate  what  the  marketplace  will  provide  of  its  own  volition.  After  mapping  this  to 
estimated  needs,  STARS  should  move  to  construct,  with  its  own  resources,  what  is  considered 
necessary  and  unlikely  to  be  provided  by  the  marketplace  itself.  Government-supported  construction 
will  prime  the  marketplace  by  convincing  suppliers  of  feasibility  and  an  established  need.  In 
addition,  STARS  should  begin  to  rely,  as  early  as  possible,  on  the  emerging  marketplaces  as  much 


as  it  safely  can.  To  do  this,  STARS  must  be  able  to  ensure  that  items  that  it  plans  on  obtaining  from 
the  marketplaces  are  available  when  needed.  This  requires  forecasting  DoD  software  technology 
requirements  and  knowing  the  extent  to  which  industry  will  be  able  to  meet  these  requirements 
through  its  own  activities. 

Even  if  the  suggested  marketplace  theme  is  not  adopted  for  the  STARS  Program,  then  this 
principle  still  has  some  applicability.  Forecasting  needs  and  estimating  future  state-of-the-an 
technology  should  be  performed  regardless  of  the  theme.  Announcing  needs  and  looking  to  external 
sources,  rather  than  attempting  to  define  and  contract  for  technology  to  meet  these  needs,  is  an 
approach  that  can  be  successfully  used  by  the  DoD  under  a  variety  of  themes. 

Capitalization  on  other  programs  and  activities.  A  second  principle  is  to  capitalize  on  other 
programs  and  activities  as  much  as  possible.  A  broad  spectrum  of  external  programs  and  activities 
need  to  be  considered.  The  STARS  Program  is  pervasive  in  that  it  will  eventually  involve  all 
communities  that  use  or  develop  software.  These  communities  include  DoD  components,  non-DoD 
Government  entities,  defense  and  commercial  industry  and  their  associations,  and  academia. 

Management  of  a  program  involving  so  many  players  will  be  complex;  it  will  demand  careful 
consideration  of  the  interrelationships  and  needs  of  these  parties.  The  Council  of  Defense  and  Space 
Industry  Associations  (CODSIA)  had  a  similar  need  to  consider  the  total  community.  Their  task, 
relating  to  reports  13-82  [2]  and  21-83  [3],  diagrammed  the  relationships  among  the  concerned 
parties.  Figure  7,  derived  from  these  exercises,  shows  the  interrelationships  as  they  relate  to  the 
STARS  Program. 

A  detailed  discussion  of  this  Figure  appears  in  Appendix  A  and  only  a  brief  synopsis, 
discussing  the  various  types  of  parties,  is  included  here.  The  STARS  Program's  external 
constituents  include  both  STARS  "partners"  and  STARS  product  and  technology  consumers: 

•  JLC/CRM  (Joint  Logistics  Commanders/Computer  Resources  Management) 
Software  Initiative.  This  initiative  has  developed  and  revamped  (with  the  assistance 
of  industry)  software  policy  and  software  product  standards  for  the  services. 

•  MCCR  Standards  Master  Plan.  This  plan  attempts  to  catalog  and  categorize  all  DoD 
and  non-DoD  standards  that  apply  to  MCCR.  Included  are  the  voluntary  standards 
organizations  plans,  as  well  as  the  more  official  sources  of  standards. 

•  Service  Components.  The  Service  Components  include  the  Army,  Navy,  and  Air 
Force.  These  organizations  are  the  developers  of  the  STARS  products  (by  chairing 
work  area  teams  and  by  managing  industry  contracts)  and  they  also  represent  the 
consumers  of  STARS  products  and  technology.  Included  in  Service  Components  are 
computer  resources  management,  defense  systems  acquisition,  and  operational  users. 

•  Industry  components.  The  industry  components  include  industrial 
companies, industrial  organizations,  and  professional  organizations  with  a  primarily 
non-Govemment  membership. 

•  Standards  bodies.  These  include  those  professional  and  industry  organizations  that 
define  and  evolve  standards  relating  to  software  creation  and  evolution. 

•  Technology  transfer  agents  The  SEI  is  the  primary  organization  responsible  for 
transferring  the  technology  developed  as  a  result  of  the  STARS  Program. 
Workshops  and  conferences  also  play  their  traditional  roles  in  effecting  this  transfer. 
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Focusing  on  critical  success  factors.  The  final  principle  discussed  is  to  focus  management 
attention  on  critical  factors.  Every  program  must  identify  a  set  of  critical  success  factors  and  track 
their  applicability  and  the  degree  to  which  they  are  being  met.  This  has  never  been  done  for  the 
STARS  Program.  Some  factors — such  as  the  appearance  of  standard  interfaces— can  be  inferred 
from  the  19  September  Program  Plan,  but  an  explicit  list  is  not  given  in  that  plan.  Under  a 
marketplace-based  strategy,  additional,  more-concrete  success  factors  can  be  delineated. 

Historically,  neither  the  Raleigh  Workshop  nor  any  Program  plan  to  date  has  addressed 
critical  success  factors.  At  the  Raleigh  Workshop  the  STARS  Program  consisted  of  a  set  of  nine 
areas,  each  of  which  addressed  part  of  the  "software  problem"  and  was  planned  more-or-less 
autonomously  from  the  others.  A  set  of  success  factors,  in  the  sense  of  things  that  have  to  happen  in 
order  for  the  STARS  Program  to  be  successful,  was  never  identified.  There  was  not  then  (nor  has 
there  ever  been)  a  comprehensible,  concrete  statement  along  the  lines  of:  "These  are  the  five  things 
(events,  conditions,  technical  breakthroughs,  etc.)  that  must  take  place  in  order  for  the  STARS 
Program  to  accomplish  its  goal." 

While  the  19  September  Program  plan  does  not  give  an  explicit  list  of  critical  success  factors, 
it  does  imply  a  number  of  factors  including:  create  valid,  reliable,  objective  and  public  evaluation 
criteria;  standardize  information  interfaces,  and  obtain  support  from  the  STARS  constituency, 
including  DoD  Program  Managers  and  contractors.  Any  list  of  factors  must  be  more  complete  both 
in  its  breadth  and  in  its  depth  since  concrete,  specific  factors  are  needed  for  successful  Program 
guidance  and  assessment.  These  factors  should  not  only  be  used  to  measure  progress  but  they 
should  also  be  used  to  assess  the  role  and  priority  of  various  STARS  activities. 

The  lack  of  a  clear  set  of  success  factors  reflects  an  overall  lack  of  focus  of  the  STARS 
Program.  STARS  cannot  be  everything  to  everybody.  It  must  try  to  do  a  few  things  well.  These 
things  need  to  be  identified  and  tracked  closely.  The  following  is  the  beginning  of  a  suggested  list  of 
marketplace  theme-related  success  factors: 

( 1 )  Carefully  predict  DoD  needs  in  the  1990's. 

(2)  Carefully  predict  what  the  marketplace  will  supply. 

(3)  Stimulate  market  investment  in  neglected  areas. 

(4)  Build  what  the  marketplace  will  not  supply. 

(5)  Assess  impact  of  the  STARS  Program  by  how  well  DoD  needs  are  being  met. 

These,  in  turn,  suggest  other  success  factors  at  the  next  level  of  detail,  such  as  the  ones  implied  by 
the  19  September  Program  Plan.  One  of  the  early  activities  must  be  to  examine  and  expand  this  list. 

4. 3. 2. 5. 3  Execution  Strategy 

The  previous  Section  provided  some  general  guidance  and  recommendations  for  getting  the 
STARS  Program  moving,  particularly  under  a  marketplace  theme.  Successful  implementation  will 
require  adoption  of  some  changes  in  the  current  execution  approach,  i.e.,  adoption  of  five 
fundamental  management  precepts,  and  immediate  actions  to  gain  control  and  produce  near-term 
successes.  These  actions  are  necessary  regardless  of  which  overall  theme  or  strategy  is  followed. 


Fundamental  management  precepts.  With  the  theme  of  fostering  and  stimulating  competitive 
marketplaces,  management  of  STARS  must  continually  fine  tune  its  strategy  to  adapt  to  the  evolving 
needs  of  its  constituency.  The  suppliers  and  consumers  of  tools  will  need  balancing  against  evolving 


needs  of  DoD  as  the  customer.  Programmatic  plans  and  priorities  will  require  adjustments  in  tactics, 
schedules,  and  controls. 

There  are  five  precepts  by  which  the  STARS  program  should  be  managed  and  tuned  to 
remain  responsive  to  the  needs  of  its  constituency: 

Precept  1:  There  should  be  an  annual  Strategic  Planning  Cycle.  The  purpose  of  this 
strategic  planning  activity  should  be  to  assess  performance  in  the  prior  year, 
to  evaluate  the  strategy  and  to  incorporate  proposed  changes. 

Precept  2:  A  STARS  Policy  for  acquisition  and  ownership  of  products  should 
beprepared  to  provide  the  guidelines  for  development  and  distribution  of 
marketplace  products.  The  purpose  should  be  to  enumerate  policy  with 
respect  to  planning  and  review;  define  types  of  products  to  be  developed  or 
encouraged;  determine  ownership  of  products  developed  privately,  by 
government,  or  jointly  by  ER&D;  identify  control  agents  or  agencies  for 
STARS-contracted  products;  and  identify  control  agencies  for  interfaces, 
standards,  and  other  definitions. 

Precept  3:  STARS  should  create  a  symbiotic  relationship  among  DoD,  industry 
andacademia.  The  purpose  should  be  to  foster  an  environment  where  public 
and  private  needs  are  held  in  check  and  balance  as  the  parties  work  together 
within  the  requirements  of  public  law  and  FAR's.  New  means  of  cooperative 
relationships  are  needed  to  stimulate  investment  while  protecting  the  interests 
of  participating  parties. 

Precept  4:  There  should  be  quarterly  reviews  of  all  STARS  activities.  The  purpose 

should  be  to  ensure  that  the  activities  are  providing  adequate  support  for  the 
pertinent  strategic  goals  and/or  objectives.  This  review  should  be  conducted 
by  a  panel  that  will  ensure  that  corrective  action  is  taken  when  technical  or 
policy  concerns  indicate  a  deviation  from  STARS  strategy. 

Precept  5:  Area  plans  should  be  approved  by  the  SJPO.  Area  plans  should  be  tactical 

ones  that  show  how  strategic  goal(s)  will  be  addressed.  Implementation  plans 
should  address  how  the  products  resulting  from  an  effort  will  be  installed, 
inserted  into  a  customer  base,  and  maintained. 

Immediate  actions.  The  panel  felt  it  was  very  important  that  STARS  get  into  execution 
quickly  and,  of  course,  pursue  relevant  activities. 

The  first  step  should  be  to  appoint  a  permanent  Director  at  once.  This  Director  must  be 
placed  in  a  situation  where  success  is  possible  and  must  be  given  effective  planning  and  execution 
authority.  This  might  require  programming  the  STARS  budget  as  an  OSD  element.  The  Director 
must  then  establish  a  clear  contracting  strategy  and  begin  moving  money  quickly.  The  panel 
suggests  spending  90%  of  available  funds  within  six  months.  After  getting  the  internal  house  in 
order,  the  Director  must  seek  out  strong  advocates  and  customers  and  encourage  the  Area  Teams  to 
do  the  same. 

Critical  to  stimulating  the  STARS  Program  is  direct  attention  to  developing  plans  that  lead  to 
incremental  products.  This  will  lead  to  the  earlier  appearance  of  results  with  the  valuable  side -effect 
of  demonstrating  early  progress  and  value.  It  will  also  preserve  the  flexibility  to  shift  plans  and 
schedules  in  response  to  newly  identified  needs  or  newly  available  technology. 


Incremental  product  development  is  almost  synonymous  with  evolutionary  prototyping.  It  is 
suggested  that  prototyping  be  done  early  and  often.  It  is  recognized  that  this  may  be  incompatible 
with  use  of  MIL-STD-2167  and  may  require  the  development  of  new  contracting  approaches.  It 
was  considered  important  not  to  let  these  potential  problems  steer  the  STARS  Program  away  from  a 
strong  committment  to  a  prototyping  approach. 

The  19  September  Program  Plan  reflects  an  incremental  building  approach  to  the  STARS- 
SEE  development.  While  this  is  a  move  in  the  right  direction,  it  is  felt  that  more  is  needed.  In 
particular,  there  is  still  too  much  emphasis  on  early,  Government-based  definition  of  the  eventual 
product  Prototyping  should  be  employed  to  not  only  assist  in  designing  the  STARS-SEEs  but 
should  also  assist  in  evolving  the  definition  of  their  functionality. 

4.4  STARS  Areas 

In  this  section,  each  of  the  STARS  areas  is  analyzed  as  it  was  presented  in  the  September  19 
Program  Plan  and  the  STARS  August  Quarterly  Briefings.  The  intent  is  to  assess  how  completely 
each  Area's  goals,  objectives  and  plans  are  defined  and  how  well  they  contribute  to  meeting  the 
overall  Program's  goals  and  objectives. 

Each  of  the  Areas  has  specific  goals  and  is  organized  into  activity  areas  intended  to  meet 
these  goals  To  assess  an  Area,  the  general  approach  was  to  first  map  the  Area's  goals  back  to  the 
Program  $  goals,  then  determine  how  the  Area's  activities  were  grouped  to  support  meeting  its 
goals,  and  finally  consider  the  Area's  intended  results  in  terms  of  their  completeness  and  their 
contribution  to  meeting  the  Area's  goals. 

The  panel’s  assessment  of  each  of  the  Areas  is  presented  in  this  Section.  Prior  to  these 
assessments,  the  Program's  goals  themselves  are  discussed.  In  addition  to  the  current  six  Areas,  the 
panel  considered  the  two  Areas,  Systems  and  Human  Engineering,  that  had  been  distributed  across 
the  other  Areas  subsequent  to  their  original  definition  at  the  Raleigh  Workshop. 

■t  4  !  Goals 

The  19  September  Program  Plan  specifies  a  set  of  ten  specific  goals.  Six  are  pertinent  to 
technology  development;  three  pertain  to  the  transition  of  technology  into  use;  and  one  concerns 
program  assessment. 

The  ways  in  which  these  goals  relate  to  the  Program's  Mission  are  not  discussed  in  the 
document.  The  goals  are,  however,  essentially  identical  to  those  specified  in  the  report  produced  by 
the  Goals  and  Objectives  Working  Group  (GOWG).  One  difference  is  that  whereas  the  GOWG 
report  lists  'develop  interfaces  between  environments"  as  a  separate  goal,  this  goal  is  included  as  part 
of  one  of  the  Program  Plan's  goals.  Another  difference  is  the  Plan  has  an  explicit  goal  concerning 
reusability  and  application-specific  needs — these  issues  were  not  spelled  out  in  the  GOWG  report's 
statement  of  goals,  but  were  rather  listed  as  global  issues.  A  final  difference  is  that  the  plan's 
program-assessment  goal  was  not  stated  in  the  GOWG  Report  since  program  assessment  is  a 
management  goal  whereas  the  GOWG  Report  dealt  exclusively  with  technical  goals. 

Therefore,  the  reduction  of  the  Program's  mission  to  its  specific  goals  can  be  explained  by 
using  ’.he  discussion  in  the  GOWG  report.  It  is  not  recommended  that  the  discussion  actually  appear 
m  the  Plan,  only  that  the  GOWG  report  be  made  consistent  and  used  as  a  supporting  document. 

The  goals  of  the  STARS  Program  emphasize  supporting  the  development  of  new  MCCR 
software.  A  possible  additional  goal  is  to  attend  to  the  problem  of  "renewing"  the  mountains  of 


existing  MCCR  software.  This  would  involve  supporting  the  enhancement  of  existing  software  and 
t.  e  ransformation  of  it  into  a  form  amenable  to  support  through  use  of  STARS  environments. 

4.4.2  Relationship  of  the  Areas  to  the  Program's  Goals  and  to  Each  Other 

4.4.2. 1.  Areas  and  Program  Goals 

The  STARS  Program  is  currently  divided  into  six  Areas.  The  relationships  of  these  Areas  to 
the  Program's  goals,  as  presented  in  the  19  September  Program  Plan,  are  depicted  in  Figure  8. 

While  coverage  seems  complete,  several  relationships  are  missing.  First,  the  discussion  of 
the  Areas  later  in  the  plan  indicates  some  activities  are  planned  in  support  of  overall  Program  goals, 
but  that  support  is  not  depicted  here.  For  example,  the  Software  Engineering  Environment  and 
Methodology  Areas  plan  to  develop  technology  transition  mechanisms.  Second,  some  critically 
necessary  relationships  are  missing,  both  from  this  depiction  and  the  plans  for  the  Areas  themselves. 
For  example,  the  Business  Practices  Area  should  be  planning  to  support  the  "reusability  and 
application-specific  needs"  goal  by  developing  the  motivation  and  policies  needed  to  foster  the 
practice  of  reusability. 

The  seemingly  random  relationship  between  the  Areas  and  the  Program's  goals,  reinforced  in 
Figure  8,  has  been  a  long-standing  problem.  Primarily,  it  has  led  to  an  inability  to  justify  the 
suitability  of  the  Program's  organization.  The  diagram  suggests  two  solutions.  The  less  preferable 
by  far  would  be  to  restate  the  goals  so  that  each  Area  supports  achieving  all  the  goals.  The  preferable 
solution,  compatible  with  solving  some  of  the  relationship  flaws  noted  above,  is  to  capitalize  on  the 
fact  that  there  are  really  two  types  of  goals.  First,  there  are  those  goals  that  relate  to  a  specific 
capability  needed  for  effective  support  of  the  software  creation  and  evolution  process. 
For  instance,  a  variety  of  modem  methods,  and  the  ability  to  select  among  them,  is  required. 
Second,  there  are  goals  that  pertain  to  characteristics  of  the  process  or  the  products  it  produces.  For 
instance,  reusability  is  a  characteristic  of  the  Ada  software  language  process.  It  would  seem  natural 
to  have  each  Area  responsible  for  achieving  a  specificly  needed  capability  while  at  the  same  time 
contributing  all  of  the  process  and  product  characteristics. 

4  4.2.2.  Interconnections 

A  study  of  August  1985,  Applications  Area  plans  revealed  that  many  area  activities  should  be 
linked  to  activities  of  other  areas.  Of  forty-seven  one-way  links  that  we  felt  probably  should  be 
institutionalized,  there  were  none  that  appeared  to  be  institutionalized.  There  were  six  expressly 
stated  links  and  six  weakly  stated  links.  Thirty-five  of  the  one-way  links  that  probably  should  exist 
were  not  stated  at  all.  Some  management  attention  should  be  brought  to  bear  on  this  problem. 

4.4.3.  SEE  Area 

In  this  Section,  we  analyze  the  Software  Engineering  Environment  (SEE)  Area  plan 
presented  in  the  19  September  Plan  and  the  SEE  Area  activities  as  presented  in  the  August  Quarterly 
Review.  The  panel  has  produced  an  adjusted  plan  of  action  for  the  SEE  Area  that  addresses  the 
criticisms  the  panel  has  found.  This  plan  is  presented  at  the  end  of  the  section. 

4.4.3. 1  SEE  Area  Overview 

For  the  SEE  Area,  the  19  September  Program  Plan  discusses  the  overall  STARS  Program 
approach  to  developing  the  Program  s  STARS-SEE  products.  The  plan  does  not  indicate  concrete 
goals  and  activities  that  indicate  the  SEE  Area’s  role  in  developing  these  products.  It  can  be  inferred 
that  the  SEE  Area  will  play  a  major  role  and  the  discussion  of  the  SEE  Area  here  attempts  to  make 
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these  inferences  for  the  higher  level  aspects  of  this  area.  The  details  of  activities  in  this  Area  are 
taken  from  documents  prepared  by  its  coordinating  team. 

The  general  intent  and  strategic  focus  for  this  area  reflect  the  central  role  it  plays  in  producing 
the  STARS-SEE  products.  This  area  establishes  the  basis  for  preparing  specific  environments  in 
response  to  recognized  needs  and  does  this  by  producing  environment  substrates  on  top  of  which 
tools  may  be  easily  installed. 

The  goals  for  the  Area,  as  inferred  from  the  Program  plan,  generally  support  meeting  the 
Program's  goals  but  do  need  to  be  more  specific  about  the  tinning  interrelationships  between  interface 
development  and  building  of  the  various  concrete  products. 

The  SEE  Area's  plans  are  fairly  extensively  developed  because  this  Area  has  been  actively 
working  since  early  in  the  life  of  the  STARS  Program.  The  plans  emphasize  the  delivery  of  products 
in  the  near,  medium  and  long  term  and  the  investigation  of  several  fundamental  issues.  The  central, 
focusing  activity  concerns  the  production  of  the  STARS-SEE  environments  under  the  guidance  of 
MIL-STD-2167. 

There  is  a  potential  mismatch  between  the  high-level  plans  given  in  the  Program  Plan  and 
what  is  actually  being  done  in  the  SEE  Area.  This  mismatch  particularly  concerns  the  architecture  of 
the  STARS-SEE  environments  and  the  degree  to  which  these  environments  meet  production-quality 
standards.  The  high-level  plans  need  to  be  clarified  before  it  can  be  decided  whether  or  not  a 
mismatch  exists. 

4. 4. 3. 2  SEE  Area  Mission  and  Goals 

The  mission  for  the  SEE  Area,  as  it  relates  to  the  entire  program,  is  not  explicitly  stated  in  the 
19  September  Program  Plan.  In  the  Plan’s  discussion  of  the  Program's  environment  development 
activities,  however,  two  intents  for  the  SEE  Area  are  strongly  implied.  First,  the  Area  should  lay  the 
groundwork  for  being  able  to  rapidly  establish  powerful,  integrated  environments  in  response  to 
recognized  needs.  Second,  the  Area  should  provide  a  focal  point  for  activities  both  inside  and 
outside  the  Program  that  contribute  to  developing  this  capability. 

The  mapping  of  Program  goals  to  goals  for  the  SEE  Area  can  be  determined  by  considering 
what  is  actually  planned  in  the  SEE  Area  and  determining  how  these  planned  activities  relate  to  the 
Program’s  and  Area's  goals.  Coverage  of  the  Program's  technology  development  goals  was 
considered  good.  There  are,  however,  no  planned  activities  that  address  reusability,  since  under  the 
current  organization  all  reusability  concerns  are  delegated  to  the  Application-Specific  Area. 

4  4  3  3  SEE  Area  Activities  and  Products 

Under  the  existing  SEE  Area  organization,  it  is  through  the  STARS-SEE  activity  that  the 
Area's  goals  are  met.  The  other  SEE  Area  activities  are  intended  to  be  supportive  of  the  STARS- 
SEE  activity  with  a  prototyping  activity  doing  short-term  prototyping  studies,  an  Advanced 
Environment  Concepts  activity  taking  the  longer-range  view,  and  Foundation  Studies  activities 
looking  at  some  fundamental  issues. 

The  prototyping  activity  seems  primarily  focused  on  producing  near-term,  interim  results. 
The  Advanced  Environment  Concepts  activity's  plan  is  not  well-developed  enough  to  identify  its 
products.  The  Foundation  Studies  activity  seems  to  be  a  collection  of  relatively  arbitrary  topics.  In 
short,  plans  for  the  activities  seem  to  be  incomplete  in  some  cases  and  the  ways  in  which  the 
STARS-SEE  activity  is  supported  by  the  others  seems  to  be  poorly  defined. 


4. 4.3. 4  SEE  Area  Interactions 

Interactions  within  the  SEE  Area  focus  on  the  flow  of  information  to  the  STARS-SEE 
activity  in  support  of  its  environment  construction  activity.  The  STARS-SEE  activity  is  also  the 
focal  point  for  interactions  with  other  STARS  areas,  being  primarily  the  reception  point  for  the  tools 
developed  in  the  other  areas  but  also  receiving  the  definition  of  methods  to  be  supported  from  the 
Methodology  Area  and  suggestions  regarding  instrumentation  from  the  Measurement  Area. 

The  products  of  the  SEE  Area  flow  into  the  DoD  Community  at  large.  Nothing  is  said  in 
the  Plan  about  a  flow  of  information  or  products  to  the  other  STARS  Areas.  The  current  STARS- 
SEE  activity  plans  have  not  explicitly  defined  any  interactions  with  the  other  areas  for  the  purpose  of 
affecting  the  activities  in  those  other  areas  by  the  delivery  of  SEE  Area  products. 

4.4. 3.5  SEE  Area  General  Strategy 

The  general  strategy  for  organizing  and  carrying  out  activities  in  the  SEE  Area  is  not 
discussed  in  the  19  September  Plan.  The  discussion  in  the  plan  of  the  SEE  Area  instead  addresses 
the  overall  strategy  for  developing  the  STARS-SEE's. 

As  the  Area  is  currently  functioning,  there  are  several  key  strategic  points.  First,  the  Area's 
activities  are  partitioned  into  four  parts.  Three  of  these  attend  to  obtaining  near-term,  medium  term 
and  long-term  results,  respectively.  The  fourth  addresses  several  fundamental  issues  that  affect  all  of 
the  results  Second,  all  major  environment  development  activities  are  being  carried  out  by  following 
MIL  STD-2167. 

4  4  3.6  SEE  Area  Specific  Strategy 

As  presented  in  19  September  1985  Program  plan,  the  cornerstone  of  the  STARS  program  is 
the  development  of  powerful  integrated  software  environments.  The  strategy  for  this  development 
composed  the  following:  (1)  Establishing  a  substrate  upon  which  tools  can  easily  be  installed;  (2) 
fostering  industry  participation  while  retaining  ability  to  define  directions;  and  (3)  Government 
sponsorship  of  at  least  two  proof-of -concept  substrates  and  at  least  two  full-capability  environments. 

While  basically  sound,  this  strategy  is  unclear  in  several  regards.  It  is  unclear  how  Areas 
other  than  the  SEE  Area  will  be  driven  by  this  strategy.  The  need  for  government  sponsorship  of 
two  full-capability  environments  is  unclear.  It  is  unclear  whether  the  definition  of  environments  is 
intended  to  provide  a  logical  structure  for  problem  identification,  or  a  physical  structure  for 
construction  project  definition.  The  relationship  between  the  framework  and  the  substrate  is  not 
clear;  the  plan's  diagrams  indicate  that  the  substrate  is  obtained  by  adding  generic  tools  to  the 
framework  whereas  the  text  indicates  that  the  framework  is  partially  composed  of  the  substrate. 

4.4  3.7  Comments  Regarding  SEE  Area 

The  plan  is  not  as  explicit  about  the  goals  and  activities  for  the  SEE  Area  as  it  is  for  the  other 
ureas.  For  example,  there  is  an  organization  of  activities  within  the  SEE  Area  that  is  not  presented  in 
the  Plan. 

This  raises  the  question  of  whether  there  is  a  mismatch  between  what  management  wants  and 
what  is  actually  being  done  in  the  SEE  Area.  Before  this  can  be  determined,  the  nature  of  the 
development  approach  being  suggested  by  management  needs  to  be  clarified.  This  involves 
clarifying  the  critical  concepts,  the  general  nature  of  near-term  results,  and  the  implications  of 
management  directions  for  an  environment's  architecture.  Following  this  clarification,  the  extent  to 
which  current  projects  are  compatible  with  management  direction  can  be  determined.  In  particular. 


the  issues  of  whether  the  current  projects  are  fostering  the  development  of  multiple  environments  and 
developing  the  general  interface  requirements  prior  to  implementation  can  be  addressed. 

4.4. 3. 8.  An  Alternative  SEE-Related  Strategy 

The  panel  felt  that  the  strategy  for  the  SEE-related  activities  presented  in  the  19  September 
Program  Plan  was  oriented  toward  building  environments  as  opposed  to  an  orientation  toward 
stimulating  the  marketplace  to  provide  those  environments.  The  panel,  realizing  that  some 
environment-building  activities  are  required,  developed  a  strategy  that  it  felt  was  more  consistent 
with  the  suggested  marketplace  theme.  A  key  aspect  of  this  strategy  is  to  shift  attention  away  from 
developing  complete,  wide-spectrum  environments  (at  least  under  Government  sponsorship)  and 
toward  the  production  of  compatible  toolsets.  That  strategy  is  explained  below.  The  Panel 
suggested  that  STARS  consider  this  strategy. 

The  strategic  model  suggested  by  the  panel  for  the  development  and  acquisition  of  and 
portable  rehostable  toolsets  used  in  the  development  of  portable  but  application-  specific  software  is 
based  on  extensive  utilization  of  the  software  marketplace.  The  approach  is  to  interact  with  the 
software  development  marketplace  as  an  intelligent  and  powerful  consumer.  In  order  to  stimulate 
this  sector  to  provide  appropriate  and  sufficient  responsiveness,  the  STARS  program  must  take  the 
following  four  separate  but  related  actions: 

( 1 )  To  reduce  the  risk  of  commercial  participation  by  demonstrating  technical  feasibility  through 
the  development  of  necessary  prototypes.  These  will  be  produced  privately  and  made 
available  privately. 

(2)  To  provide  standards  and  guidelines  for  the  development  of  tools,  both  prototypes  and 
production  quality,  that  will  be  used  in  the  toolsets. 

(3)  To  announce  an  intention  to  acquire  tools  developed  under  commercial  sector  support  to  be 
used  in  the  toolsets. 

(4)  To  modify  government  acquisition  regulations  and  propose  legislation,  if  necessary,  to  allow 
commercial  ownership  of  most  of  the  products  developed  under  this  strategy. 

The  actual  development  and  acquisition  would  follow  a  simple  tactical  plan  with  three 
principal  activities.  These  activities  are: 

o  First,  a  small  number  of  contracts  (two  or  three)  would  be  initiated  to  construct  a 
distributed  CAIS  kernel  using  an  existing  virtual  machine  (e.g.,  VMS,  UNIX,  CMS,  MS- 
DOS,  Rational,  etc.)  as  a  basis.  We  call  the  results  of  these  contracts  "backplanes.” 

o  Second,  building  upon  the  resulting  backplanes,  four  to  six  additional  contracts  would  be 
initiated  to  construct  prototype  generic  tools  and  tool  fragments.  These  prototypes  would 
be  hosted  on  the  backplane  in  much  the  same  manner  as  hardware  circuit  boards  are 
inserted  into  an  hardware  backplane.  This  will  require  standardization  beyond  that 
provided  CAIS.  A  core  set  of  typ^d  objects  (processes/data  files)  must  be  specified. 

o  Finally,  a  larger  number  of  additional  contracts  would  be  initiated  to  construct  application- 
specific  prototypes  that  would  also  be  hosted  on  the  backplanes.  Thus,  the  resulting 
families  of  application-specific  toolsets  would  be  as  compatible  as  possible  since  they  all 
use  the  same  underlying  model  but  would  only  carry  the  tools  that  are  appropriate  to  their 
application  domain. 
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This  approach  to  producing  compatible  toolsets  tuned  to  various  application  areas  is  likely  to  produc  e, 
higher  quality  results,  with  a  shorter  delivery  time,  than  the  existing  STARS  SEE  strategy. 

4.4.4  Methodology  Area 

The  Methodology  Area  within  the  STARS  Program  addresses  the  problem  of  providing  a 
variety  of  software  creation  and  evolution  approaches  for  use  on  DoD  software  projects.  The  overall 
intent  is  to  broaden  both  the  applicability  of  the  collection  of  methods  available  for  use  on  DoD 
software  projects  and  the  extent  to  which  they  are  actually  employed. 

4.4.4  1  Methodology  Area  Mission 

The  STAR'S  Methodology  Area  focuses  on  the  principles,  practices  and  procedures  used  for 
the  creation  and  evolution  of  software.  Its  scope  is  broad,  encompassing  both  technical  and 
managerial  concerns. 

The  overall  intent  is  to  increase  the  breadth  of  the  collection  of  methodologies  available  to 
support  DoD  software  projects,  the  extent  of  automated  support  for  these  methodologies,  and  the 
extent  of  their  use  throughout  the  DoD  community.  Attention  is  directed  in  particular  to  those 
methodologies  supporting  the  use  of  the  Ada  language  and  software  reusability. 

4.4.4  2  Methodology  Area  General  Strategy 

The  Area's  strategy  is  to  focus  on  the  problems  surrounding  the  effective  use  of  modem 
methodologies  and  let  that  drive  expansion  of  the  collection  of  available  methodologies.  The 
underlying  premises  are:  that  current-day  methodologies  are  adequate  (but  limited);  that  no  single 
methodology  will  be  sufficient  even  for  an  average  set  of  software  projects;  and  that  the  major 
inhibitor  to  effectively  using  modem  methodologies  is  an  inability  to  identify  which  methodology  to 
use  for  a  specific  project. 

According  to  this  strategy,  in  the  mainstream  of  activity  is  the  development  of  a  capability  to 
choose  a  methodology,  given  the  needs  of  a  specific  project.  The  two  major  steps  in  this  choice 
would  be:  (1)  an  initial  evaluation  of  alternatives  followed  by  (2)  a  selection  among  those  judged  to 
be  acceptable.  The  Methodology  Area's  activities  seek  to  support  both  evaluation  and  selection, 
particularly  through  the  development  of  materials  that  support  a  heuristic  approach. 

Through  these  activities,  the  requirements  for  methodologies  specific  to  DoD  problems  will 
be  identified.  These  requirements  will,  in  fact,  form  the  basis  for  evaluation  and  selection  criteria. 
In  developing  the  requirements,  particular  attention  will  be  given  to  supporting  the  use  of  the  Ada 
language  and  software  reusability. 

Development  of  the  evaluation  and  selection  capabilities  will  also  help  identify  flaws  or 
incompleteness  in  the  methodologies  available  today.  The  Methodology  Area  will  attempt  to  correct 
the  situation  by  enhancing  existing  methodologies  when  possible,  and  developing  new  ones  when 
necessary. 

4.4  4.3  Methodology  Area  Goals 

The  goals  for  this  Area  are: 

(1)  To  develop  -  the  capability  of  evaluating  methodologies  with  the  intention  of  selecting 
among  alternatives. 
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(2)  To  identify  existing  methodologies  of  particular  value  for  DoD  projects  and  refining 
them,  as  needed,  to  enhance  their  applicability. 

(3)  To  conduct  methodology  evaluation  experiments. 

(4)  To  develop  automated  tools  supporting  various  methodologies. 

(5)  To  assure  that  the  STARS-produced  environments  incorporate  the  methodology  support 
tools. 

(6)  To  foster  widespread  use  of  the  methodologies  through  recommendation,  support  and 
active  promotion. 

These  Methodology  Area  goals  primarily  support  meeting  the  STARS  Program’s  technical 
goals.  By  assuring  a  wide  breadth  of  coverage  for  the  Program's  technical  goals,  one  can  assume 
adequate  support  for  meeting  those  goals. 

There  is  one  flaw  however— the  Area's  aim  of  developing  a  methodology  selection  capability 
is  not  reflected  in  the  statement  of  the  Program's  goals.  This  flaw  should  be  corrected  by  expanding 
the  scope  of  the  Program's  goal  to  "provide  a  variety  of  modem  methods"  to  encompass  a  way  of 
providing  the  means  to  select  among  those  methods. 

4.4.4.4  Methodology  Area  Activities,  Results  and  Interactions 

The  Methodology  Area  is,  in  general,  adequately  structured  to  meet  its  goals.  The  activities 
are  partitioned  into  four  activity  areas  and  an  extensive  variety  of  products  have  been  identified  for 
each  activity  area 

The  19  September  Program  Plan  mentions  a  few  interactions  among  the  activity  areas  and 
among  the  Methodology  Area  and  the  other  STARS  Areas  as  well  as  outside  the  STARS  Program 
itself.  The  most  well  defined  of  these  interactions  are  those  outside  the  Program.  The  least  well- 
defined  are  those  with  the  other  STARS  Areas. 

The  interactions  within  the  Methodology  Area  and  among  it  and  other  STARS  activities  needs 
extensive  work.  In  particular,  the  flow  of  information  and  products  into  the  the  Methodology  Area 
has  not  been  defined  at  all. 

4.4.4  5  Comments  Regarding  Methodology  Area 

In  general,  the  Methodology  Area  is  well  constituted.  Its  goals  are  reasonable  ones  that 
adequately  serve  to  meet  the  Program's  technical  goals.  Its  anticipated  activities  should  provide  an 
adequate  basis  for  useful  near-term  and  long-range  results. 

Some  of  the  flaws  in  the  definition  of  this  area  and  its  activities  have  already  been  noted:  the 
STARS  Program's  method-oriented  goal  should  reflect  the  Methodology  Area's  selection-capability 
activity  and  the  interactions  within  the  Area  and  among  the  Area  and  other  STARS  Areas  are  not 
sufficiently  well  defined.  In  addition,  the  Area's  plans  are  incomplete.  Long-range  plans  are  weak  or 
nonexistent.  For  the  methodology/tool  development  and  the  insertion  activities,  no  concrete  short¬ 
term  plans  are  presented.  For  the  methodology/tool  development  activity,  no  coherency  or  logic  is 
present  for  the  topics  identified  as  those  that  should  be  emphasized. 
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4.4.5  Measurement  Area 


In  the  following  sections,  the  Measurement  Area  is  analyzed  from  two  perspectives.  First,  it 
is  looked  at  from  within  the  context  of  a  "build-it"  approach,  in  which  the  objective  of  the  STARS 
Program  ts  to  provide  production-quality  tools,  specifically  in  the  form  of  the  STARS-SEEs.  The 
Measurement  Area  is  then  discussed  within  the  context  of  a  marketplace-stimulation  approach,  in 
which  the  objective  of  the  Program  is  to  stimulate  investment,  competition  and  innovation  within  the 
private  sector,  with  STARS  building  only  that  which  the  marketplace  will  not  provide. 

4  4.5. 1  The  Measurement  Area  From  a  Build-It  Perspective 

The  Measurement  Program  Plan  (4  Nov  85)  presents  a  "build-it"  strategy  for  promoting 
measurement.  This  has  been  the  approach  taken  by  the  Measurement  Area  from  the  earliest  planning 
stages  of  STARS  (pre-Raleigh)  Even  at  that  point  in  time,  the  lack  of  consistent,  interpretable,  and 
useful  measures  and  models  was  recognized  as  a  problem  that  cut  across  all  areas.  The  Measurement 
Area  was  intended  to  fulfill  two  broad  functions.  The  first  was  to  provide  expertise  to  the  other  area 
in  identifying  measures  and  in  constructing  and  validating  models.  The  second  function  was  to 
provide  the  basis  for  a  quantitative  assessment  of  the  impact  of  the  STARS  Program.  The 
Measurement  Area  has  maintained  continuity  with  the  earlier  planning  efforts.  The  most  recent 
Measurement  Program  Plan  includes  activities  to  define,  measures,  develop  models  and  production 
quality  measurement  tools,  provide  demonstrations  of  the  use  of  measurement  in  the  software  life 
cycle,  and  develop  the  necessary  training  material  and  seminars  to  aid  in  the  technology- insertion 
process. 

Failure  to  foster  use.  The  biggest  problem  with  the  plan  for  the  Measurement  Area,  as  with 
the  entire  build-it  approach,  is  that  it  does  not  address  the  fundamental  reasons  for  why  measurement 
and  modeling  are  not  commonly  applied.  Simply  building  more— whether  more  measures,  more 
models,  more  training  material— does  not  address  the  problem.  Much  more  already  exists  in  terms 
of  models  and  measures  than  is  ever  being  used.  Clearly,  something  more  fundamental  is  missing 
than  the  lack  of  technology.  That  is  what  the  marketplace-stimulation  approach  addresses  in  an 
attempt  to  provide  incentives  to  apply  the  existing  technology  in  a  way  that  is  meaningful  and 
innovative. 

Lack  of  clear  goals  for  the  area.  Whether  one  looks  at  the  1 9  September  Program  plan  or  ai 
the  Measurement  Program  Plan  (4  November  1985),  there  is  a  lack  of  clear,  crisp  goals  for  the  area. 
Rather,  the  goals  appear  to  be  so  broad  as  to  be  meaningless.  The  STARS  Program  plan  lists  two 
goals:  "(1)  The  development  and  adoption  for  use  of:  quantifiable  indices  of  merit;  procedures 
supporting  evaluations  and  comparisons  of  software  products;  and  assessment  of  the  processes 
associated  with  software  development,  life  cycle  evolution,  and  field  use;  (2)  The  development  of 
means  for  assessing  how  well  the  STARS  Program  is  meeting  its  goal." 

The  "Goals"  section  of  the  Measurement  Area  plan  lists  only  one  very  high-level  goal  - 
to  develop  measurement  technology  which  can  provide  the  data,  models,  and  information  to 
support  software  acquisition  and  software  engineering  decision-making  for  better  planning  and 
specification,  for  improving  software  quality  and  performance,  and  for  reducing  life  cycle  software 
costs".  Clearly,  the  goals  for  the  area  need  to  be  further  decomposed  and  made  more  precise. 

Lack  of  interaction  with  other  areas.  Another  problem  is  the  lack  of  interaction  with  the  other 
areas.  The  plan  alludes  to  such  interaction,  but  there  are  no  concrete  plans  for  carrying  it  out.  If  it  is 
to  be  effective,  Measurement  cannot  exist  as  an  isolated  activity.  Its  original  purpose  was,  and  its 
current  one  must  be,  to  support  the  other  areas.  The  attempt  to  isolate  measurement  as  a  separate 
activity  is  probably  the  reason  for  the  lack  of  clear-cut  goals  as  discussed  above,  without  being  tied  to 
the  other  areas,  it  is  trying  to  do  everything. 
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The  suggested  approach  to  assessing  impact  of  STARS  is  confused.  Both  under  the  build-it 
approach  and  under  the  marketplace-stimulation  approach,  a  very  important  function  of  the 
Measurement  Area  is  to  assess  the  impact  of  the  STARS  Program.  This  needs  to  be  given  far  more 
thought  than  it  has  received  to  date.  The  approach  mentioned  in  the  Measurement  Area  Plan,  which 
is  to  compare  projects  before  and  after  the  infusion  of  STARS-developed  technology,  is  a  confused 
one,  with  time  being  the  confusing  factor.  Instead,  STARS  and  non-STARS  projects  should  be 
compared  at  similar  points  in  time. 

The  Measurement  Area  plan  is  disjoint.  As  a  final  note,  the  plan  is  disjoint,  with  no  clear 
relationship  between  its  first  and  second  halves.  For  example,  the  First  part  (under  "Scope") 
mentions  a  framework  for  characterizing  MCCR  systems  in  terms  of  eight  dimensions  but  that 
framework  is  not  mentioned  again.  There  are  planned  projects  mentioned  under  "Scope"  that  do  not 
appear  later  in  the  "Approach"  Section. 

4. 4.5. 3  An  Analysis  of  the  Measurement  Area  From  a  Marketplace-Stimulation  Perspective 

The  current  Measurement  Area  plan  can  be  summarized  by  pointing  out  that  there  are  useful 
and  needed  activities  in  the  plan,  especially  those  involved  in  providing  standard  definitions  for 
various  measures  and  in  developing  a  central  database  of  measurement  data.  Activities  such  as  tool 
building,  model  development  and  calibration,  and  the  development  of  training  materials  are  unlikely 
to  increase  the  use  of  quantitative  techniques  because  the  necessary  incentives  are  missing.  The 
marketplace-stimulation  theme  has  been  suggested  as  a  way  of  providing  these  incentives  in  the 
following  section.  The  Measurement  Area  is  analyzed  from  that  perspective  section. 

Measurement  can  play  a  pivotal  role  in  the  marketplace-stimulation  strategy  because,  to  the 
extent  that  DoD  requirements  can  be  expressed  and  evaluated  quantitatively,  they  become  objective 
and  precise.  The  more  vague  the  characteristic  being  specified,  the  more  important  it  is  that  it  be  tied 
to  something  measurable.  For  example,  a  seemingly  fuzzy  characteristic  such  as  "easy  to  use"  can 
be  precisely  specified  by  requiring  that  a  representative  sample  of  users  be  able  to  accomplish  a  given 
task  within  a  given  amount  of  time  with  fewer  than  a  given  number  of  errors. 

The  Measurement  Area  can  play  an  essential  role  in  the  Program  by  ensuring  consistency  and 
uniqueness  among  the  measurement  activities  in  the  various  areas.  It  is  essential  that  measures  such 
as  lines  of  code  be  defined  and  calculated  in  a  consistent  way.  Progress  in  software  measurement 
has  suffered  greatly  from  the  lack  of  such  continuity.  The  Measurement  Area  can  also  serve  a  useful 
function  in  assessing  the  impact  of  the  STARS  Program.  Exactly  how  this  will  be  done,  however, 
requires  a  further  study. 

4.4.6  Application  Specific  Area 

The  following  findings  are  derived  from  the  August  1985  Quarterly  Review  documents. 
They  reflect  panel  concerns  and  should  encourage  management  attention  to  determine  if  the  concerns 
are  justified  and  whether  they  would  be  alleviated  or  aggrevated  by  FY86  plans. 

The  panel  found  that  virtually  all  responsibility  for  supporting  reusability  of  software  resides 
in  the  Application-Specific  Area.  There  are  aspects  of  reusability  that  require  attention  from  the  other 
areas  as  well.  In  particular,  an  opportunity  is  being  missed  not  to  strongly  direct  the  SEE  Area 
toward  reusability.  To  evolve  the  STARS-SEE's  as  planned  by  management,  the  SEE  will  have  to 
develop  a  good  deal  of  reusability-supporting  technology. 


Some  support  for  almost  all  high  level  STARS  Program  goals  is  provided  by  this  Area. 
Some  of  the  goals,  however,  are  only  weakly  supported.  The  goal  of  supporting  program 
assessment  currently  appears  unaddressed.  The  goals  of  understanding  and  providing  good  human 
interfaces  is  supported  as  a  minor  point  in  only  one  activity,  D5--ABICS. 


The  activities  of  this  Area  are  largely  unrelated  to  each  other  and  are  disconnected  from  the 
other  STARS  Areas.  This  lack  of  internal  interrelationships  is  not  surprising  and  may  be  justified 
by  the  need  to  demonstrate  STARS-supported  technology  in  diverse  applications.  There  are  some 
common  aspects  to  the  activities  but  it  is  not  clear  if  there  is  one  activity  that  will  coalesce  the 
common  experiences  and  adequately  transfer  the  new  knowledge.  The  schedule,  for  example,  of  the. 
"Reusability  Guidebook"  does  not  appear  to  be  taking  as  input  the  experience  gained  in  the  othet 
projects.  It  is  very  important  that  the  links  of  this  area  to  other  STARS  areas  and  other  Programs  and 
activities  be  institutionalized. 

For  example,  the  connections  of  application-specific  activities  with  the  SEE  area  are  unclear. 
Another  example  is  that  the  relationship  with  the  Business  Practices  Area  is  not  well  established 
The  Application  Specific  Area  should  work  with  the  Business  Practices  Area  to  identify  application 
areas  for  which  unique  business  practices  will  have  to  be  developed.  The  Business  Practices  Team 
should  be  responsible  for  developing  these. 

An  activity  from  which  DoD  could  benefit  a  great  deal  that  would  naturally  fall  within  this 
area  would  be  the  development  or  encouragement  of  a  standard  family  of  real-time  kernels.  The 
strategy  used  could  be  similar  to  the  proposal  herein  for  the  SEE  area  (see  Section  4.4.3. 8).  Another 
activity  that  has  been  ignored  is  the  creation  of  a  process  whereby  new  application  areas  are  brought 
into  the  new  STARS-inspired,  DoD  practice. 

4.4  6. 1  Application  Specific  Area  Goals  and  Objectives 

According  to  the  September  19  Plan  and  the  August  1985  quarterly  review  briefings,  the 
Application  Specific  Area's  goals  and  objectives  support  all  the  Program  goals  except  " motivation ' 
and  program  assessment.  The  rationale  for  the  objective  of  developing  specialized  simulators 
appears  to  be  a  bottom-up  rationalization.  It  is  unclear  why  the  common  library  effort  is  here  rather 
than  in  the  SEE  Area.  The  concept  of  technology  transition  by  "spreading  the  word"  is  not  well 
thought  out.  Otherwise,  the  objectives  are  reasonable.  The  overriding  problem  is  the  nonexistence 
of  links  to  other  areas,  especially  the  SEE  Area. 

4  4  6  2  Application  Specific  Area  Objectives,  Activities  and  Products 

Ail  objectives  are  supported  to  some  extent  by  activities.  The  panel  does  not  understand  how 
spread  the  word"  is  supported  nor  what  the  "conversion  to  Ada"  objective  supports.  We  also  do 
not  see  evidence  of  the  realization  of  the  connection  between  this  objective  and  similar  activities  in  the 
WIS  program.  The  activities  are  largely  unrelated,  but  that  is  reasonable  for  this  area.  There  should, 
however,  be  an  effort  at  extracting  the  common  experience  derived  from  these  activities.  We  also 
feel  that  the  objective  of  developing  reusable  run-time  parts  would  be  better  supported  by  an  activity 
for  demonstrating  a  family  of  (or  a  generic)  standard  real-time  kernels. 

4.4  7  Business  Practices  Area 

The  objective  of  the  Business  Practices  Area  as  stated  in  the  Quarterly  Review  of  November 
1985  is  to:  "Develop  and  introduce  concepts,  tools  and  techniques  to  improve  software  acquisition 
practices  across  program  phases.  Specifically:  software  maturity  and  reuse;  expert  management 
system  and  contract  documentation;  estimating  and  control  metrics;  DoD  software  standards;  and 
technical  and  engineering  management  support." 

The  funding  profile  is  $1  7M  in  FY86,  $5.3M  in  FY87,  and  $14. 3M  in  FY88  for  the 
following  tasks:  software  maturity  and  system  called  SWAPX  (C2);  estimating  and  control  metrics 
(C3);  standards  (C4);  support  called  TEMS  (C5) 


The  DoD  has  as  its  most  effective  mechanism  for  technology  development  and  insertion,  its 
behavior  in  the  industrial  sector  as  an  intelligent  consumer.  The  original  purpose  of  the  Business 
Practices  Area,  as  presented  at  the  initial  public  review  held  at  Raleigh,  was: 

(1)  To  develop  the  body  of  knowledge  necessary  for  the  DoD  to  act  in  the  role  of  intelligent 
consumer. 

(2)  To  develop  a  suite  of  program  management  tools  to  provide  consistency  and  support 

across  both  program  phases  as  well  as  across  programs  and  application  areas. 

(3)  To  propose  changes  to  DoD  regulations,  practices  and  standards  appropriate  to 
enhancingthe  DoD  position  in  the  business  environment.  Thus,  the  Business  Practices 
Area  should  be  concerned  with  all  the  marketplace  issues  described  in  Section  4.3.2. 

However,  the  Business  Practices  Area  task  has  narrowed  in  focus,  as  stated  above,  to  the 
acquisition  of  software  and  eliminated  any  emphasis  on  developing  the  knowledge  to  improve  and 
exploit  the  DoD  position  in  the  commercial  sector. 

The  Business  Practices  Area  has  two  subtasks,  maturity  (Cl)  and  metrics  (C2).  These 
subtasks  are  direct  linkages  to  the  Measurement  Area  yet  there  is  no  such  linkage  described  in  the 
Business  Practices  Area  plan. 

No  program  management  plan  is  in  place,  although  it  is  proposed  that  the  support  contractor 
aid  in  the  development  of  one  by  third  quarter  FY86.  It  is  apparent  that  developing  a  management 
pian  in  this  manner  will  lead  to  a  justification  of  subtasks  C1/C4.  While  it  is  critical  that  some  work 
be  performed  toward  meeting  program  objectives,  there  should  be  some  concern  about  the  subtasks 
driving  the  development  of  the  plan. 

There  is  no  linkage  to  other  task  areas,  with  the  possible  exception  of  the  measurement  area. 
STARS  must  exploit  the  fact  that  the  Business  Practices  Area  is  the  most  powerful  tool  the  STARS 
program  has  in  its  mission  the  insertion  of  appropriate  software  technology  into  the  DoD  software 
development  community. 

4.4.7. 1  Recommendations  for  Business  Practices  Area 

The  focus  of  the  Business  Practices  Area  should  be  broader.  Specifically,  the  area  should 
include: 

(1)  Developing  cost/benefit  templates  for  other  technical  areas  with  respect  to 
commercial/DoD  marketplace  effects 

(2)  Developing  an  understanding  of  the  dynamics  of  the  software  marketplace 

(3)  Understanding  the  impact  of  DoD  standards,  their  feasibility  and  cost  effectiveness  as 
well  as  the  tradeoff  between  inhibition  of  technology  insertion  and  logistical  needs 

(4)  Understanding  future  DoD  software  needs  and  likely  commercially  available  technology 

In  addition,  the  Business  Practices  Area  should  investigate  possible  altertion  of  FAR,  DFAR 
and  any  other  applicable  regulations  and  laws.  It  should  also  work  with  the  Human  Resources  Area 
to  attempt  to  alter  the  practices  of  Program  managers  in  order  to  effect  the  change  of  DoD  into  an 
intelligent  consumer. 

The  Area  should  closely  integrate  its  activities,  planned  and  ongoing,  with  the  other  areas. 


The  Area's  plan  should  be  refined  before  any  tools  are  developed.  In  particular,  the  content 
of  the  knowledge  bases  will  most  likely  change  once  an  understanding  of  the  DoD  role  in  the 
commercial  sector  has  been  developed. 

The  Business  Practices  Area  team  should  be  populated  with  industrially  knowledgeable 
members  including  lawyers  and  private  sector  executives  familar  with  the  DoD  and  the  various 
procurement  and  acquisition  strategies. 

It  should  be  recognized  that  the  most  important  subtask  in  the  STARS  effort  for  technology 
insertion,  i  e.,  getting  the  STARS  tools  used,  is  the  Business  Practices  Area.  In  addition,  it  is  the 
most  difficult  area  in  which  to  accomplish  anything  significant.  The  funding  does  not  reflect  this  and 
should  be  increased. 

4  4  8  Human  Resources  Area 

The  Human  Resources  Area’s  goals  derive  mainly  from  three  STARS  Program  goals— better 
work  force,  technology  insertion,  and  measurement.  Measurement  of  personnel  is  an  important,  but 
sometimes  overlooked,  aspect.  The  efforts  in  this  area  are  divided  into  three  activities.  The  first  two 
try  to  establish  the  size  of  the  current  work  force  and  what  should  it  be  in  the  1990  s  And  the  third 
concerns  itself  with  development  of  material  and  technology  for  education  and  training. 

If  STARS  is  to  influence  the  composition  of  the  work  force  in  the  early  1990’s,  it  needs  to 
know  what  to  aim  at  before  1988.  Some  progress  could  be  made  now  based  on  the  general  need  for 
upgrading  that  work  force. 

The  actions  required  to  impact  the  work  force  are  not  well  defined.  Some  of  this  is 
understandable  since  the  requirements  are  not  yet  established.  But  the  possibilities  and  their  expected 
costs  and  effects  should  be  enumerated.  This  is  an  area  where  the  costs  are  not  well  known  and  an 
awareness  of  what  would  be  reasonable  is  needed  immediately. 

The  lack  of  a  measurement  emphasis  on  individuals  or  organizations  could  undermine  future 
attempts  to  inspire  improvement  and  competence  through  acquisition  requirements  and  evaluations 
CAI  is  an  area  npe  for  DoD  benefit  from  the  standardization  of  such  things  as  authoring 
languages/systems. 

4.4.8. 1  Recommendations  for  Human  Resources  Area 

Plans  in  this  area  are  too  poorly  developed  for  present  implementation.  Rough  estimates  of 
the  situation  and  future  requirements  should  be  made  soon  and  then  refined  as  better  data  are 
developed.  Alternative  actions  should  be  laid  out  in  some  depth  so  that  their  costs,  difficulties  and 
impacts  can  be  understood. 

These  in  tum  should  be  used  to  establish  out-year  budgets  for  this  Area  on  a  rational  basis. 
Possibly  this  area  should  have  significantly  more  money  than  it  is  now  budgeted,  but  this  is  difficult 
to  assess  now  because  possible  future  actions  are  so  nebulous. 

The  Ada  and  STARS  Programs  need  to  establish  their  Human  Resources-related  boundaries, 
and  the  SEI  needs  to  coordinate  with  them.  For  example,  STARS  funds  might  help  buy  the  rights  to 
existing  curricula  for  inclusion  into  the  SEI  master  curriculum. 

Short  or  mid-term  results  such  as  scholarship  programs,  career  path  recommendations,  rapid 
concurrent  adaption  of  the  SEI  curriculum  for  industry  and  Government  use,  even  seminars  for 


government  employees  might  be  beneficial.  These  measures  would  also  have  the  effect  of  being 
visible  STARS  results. 

STARS  should  not  be  a  "don't  rock  the  boat"  progam.  Present  Human  Resources  Area 
plans,  possibly  because  of  their  nebulousness  and  the  low  area  budget,  appear  to  lack  boldness. 
However,  a  low  budget  may  necessitate  cleverness.  As  a  possible  example  of  an  innovative  move, 
consider  (in  line  with  the  marketplace  theme  for  STARS)  the  development  of  meaningful,  operational 
"specifications"  for  the  human  resources  DoD  acquires  on  contracts.  This  would  include  certification 
testing  of  the  various  types  of  software-related  personnel  and  topic  testing  such  as  knowledge  of 
DoD-STD-2167  or  Ada. 

4.4  9  Systems:  A  Lost  Area 

The  panel  felt  the  content  of  this  original  STARS  area  was  of  increasing  importance  to  DoD 
and  should  be  adequately  covered.  A  new  "Systems"  initiative/program  bridging  STARS  and 
VHS1C  may  be  appropriate.  In  any  case,  STARS  should  include  software  related  systems  concerns. 
For  example,  the  emphasis  on  reliability/fault  tolerance  in  the  original  Systems  area  should  not  be 
lost. 

Further,  there  is  evidence  that  industry  supports  the  idea  that  DoD  should  consider  a  systems 
approach  to  the  MCCR  engineering  problem.  For  examples,  see  the  CODSIA  reports. 

4.4.10  Human  Engineering  Area 

Human  Engineering  was  one  of  nine  task  areas  at  the  time  of  the  Raleigh  workshop,  giving  it 
considerably  more  visibility  that  it  now  has  within  the  STARS  Program.  As  originally  envisioned,  it 
was  broad  in  its  concerns,  covering  not  only  human  engineering  of  the  software  support 
environment  (including  methods  as  well  as  tools)  but  also  human  engineering  of  MCCR  systems  for 
the  end  user.  The  basic  plan  for  the  Human  Engineering  area  was  endorsed  and  refined  at  Raleigh 
and  provided  a  solid  starting  point  for  more  detailed  planning. 

Human  Engineering  as  a  separate  area  disappeared  from  the  STARS  Program  in  1984.  There 
is  now  a  tri-service  committee  addressing  human  engineering  within  the  SEE  Area,  although  it  seems 
to  have  been  largely  ignored  in  all  earlier  plannin).  The  Methodology  Area  has  paid  lip  service  to 
including  Human  Engineering  methodologies  but  has  done  nothing  concrete  to  date  nor  does  it 
currently  employ  members  with  the  required  expertise. 

4  4. 10  1  Technical  and  Managerial  Recommendations  for  Human  Engineering  Area 

We  recommend  that  the  Human  Engineering  Area  should  be  reinstated  to  focus  on  the 
technical  issues  involved  in  specifying,  evaluating,  and  selecting  products  on  the  basis  of  usability 
(and  not  on  building  production-quality  tools). 

The  strategy  for  Human  Engineering  (as  part  of  the  strategy  of  the  STARS  Program  as  a 
whole)  should  involve  being  very  clear  about  the  desired  end  products  and  how  they  will  be 
evaluated.  As  much  as  possible,  the  participation  of  high-caliber  people  with  human-factors 
experience  should  be  solicited  and  encouraged. 

STARS  should  take  steps  to  achieve  much  greater  visibility  and  support  among  all  of  its 
constituents.  The  Raleigh  workshop  served  to  focus  the  attention  of  several  nationally-known 
human-factors  experts  on  the  STARS  Program.  The  interest  and  involvement  of  people  with  these 
qualifications  can  result  in  enormous  contributions  to  the  STARS  Program.  The  reliance  on 
government  committees,  particularly  as  vehicles  for  planning,  has  clearly  not  been  successful. 


5.0  SYNOPSIS  OP  RECOMMENDATIONS 


The  panel  has  identified  and  discussed  in  the  preceding  parts  of  this  report  a  number  oi 
managerial,  thematic,  and  strategic  recommendations  for  STARS.  The  primary  recommendations  are 
summarized  in  this  Section,  along  with  a  short  rationale  for  each. 

The  most  crucial  requirement  is  to  identify,  hire,  and  empower  a  strong,  permanent  Director 
and  to  establish  a  tenable  political  situation  for  that  Director.  The  current  contention  between  OSD 
and  the  Joint  Logistics  Commanders  (JLC's)  over  who  is  in  control  of  the  Program  must  be  resolved 
quickly  if  the  Program  is  to  be  saved.  The  Director  must  institute  top-down  planning  and  control.  In 
order  to  do  this,  a  hierarchical  organization  of  dedicated  billets,  responsible  to  the  Director,  must  be 
formed.  There  are  many  acceptable  devices  that  DoD  can  use  to  accomplish  this,  but  the  end  result 
must  be  that  review  and  control  over  line  and  non-line  personnel  rest  with  dedicated  STARS 
management.  The  STARS  Director  must  also  be  given  effective  spending  and  contracting  authority. 
A  check-and-balance,  distributed,  consensus-oriented  approach  to  spending  authority  is 
inappropriate  here. 

The  panel  felt  strongly  that  STARS  should  adopt,  and  consistently  operate  within,  a  theme 
of  marketplace  stimulation.  STARS  needs  to  manipulate  the  demand  side  of  the  marketplace  by 
iurmng  DoD  into  an  intelligent  customer  for  software  technology  and  for  mission  critical  software 
STARS  must  also  stimulate  the  supply  side  of  the  marketplace  by  creating  conditions  conducive  to  a 
community  competing  to  supply  software  engineering  tools,  methods,  training,  service,  MCCR 
software  components,  and  the  other  artifacts  and  practices  STARS  and  others  recognize  as  critical  to 
meeting  DoD's  MCCR  software  requirements.  In  addition,  STARS  needs  to  help  drive  the  suppliers 
of  that  software  to  a  more  efficient  state  of  practice. 

This  is  an  inherently  more  difficult  theme  than  one  in  which  STARS  simply  pays  for  the 
development  of  new  tools,  systems,  and  related  technology  without  considering  how  these 
developments  will  be  used  or  how  they  will  reduce  costs  in  the  future.  Every  project  undertaken  by 
STARS  will  face  the  difficult  decision  of  how  much  STARS  must  fund  in  order  to  ensure  that  a 
given  capability  will  appear  in  the  marketplace  at  the  appropriate  time.  Must  feasibility  demonstration 
tie  performed9  Must  the  value  of  a  proposed  product  to  its  potential  users  be  demonstrated?  What 
will  the  measures  of  value  be?  How  will  these  measures  be  made?  Must  certain  items  be  produced 
or  purchased9  What  standards  are  required  and  when9  What  new  business  practices  within  DoD  are 
needed  to  establish  the  consumer  or  producer  sides  of  the  marketplace?  These  kinds  of  decisions 
will  require  close  cooperation  with  industry  and  will  also  necessitate  understanding  of  a  wide  range 
of  political,  entrepreneurial,  and  technical  issues  from  within  the  STARS  leadership. 

The  panel  also  recommends  that  STARS  management  take  a  new  look  at  the  set  of  areas  and 
the  relative  emphasis  the  program  places  in  those  areas.  The  panel  recommends  in  particular  that  the 
lost  areas  of  Human  Engineering  and  Systems  be  reconstituted  in  some  way.  It  may  be  that  there 
should  be  components  of  these  two  concerns  added  to  some  or  all  current  areas.  The  committee 
definitely  felt  that  linkages  between  the  Measurement  Area  and  the  other  areas  must  be  very  quickly 
arid  strongly  established  or  that  other  steps  be  taken  to  assure  concentrated  attention  to  measurement 
issues  within  each  of  the  areas. 

Having  reviewed  the  areas  and  their  relative  levels  of  emphasis,  STARS  Management  should 
review  the  constitution  of  the  area  teams  and  repeat  this  review  periodically.  The  focus  of  the  teams 
should  be  changed  from  a  management  to  a  technical  orientation  because  management  should  be 
handled  at  higher  levels.  The  teams  should  then  be  made  to  evolve  as  the  needs  of  the  Program 
change.  For  example,  it  may  be  that  as  technical  goals  are  achieved,  a  team  might  be  oriented  toward 
inserting  new  technology  into  the  Services  instead  of  developing  the  technology.  Different  people 
will  be  required  as  needs  change. 


The  panel  felt  that  measurable  goals,  objectives  and  success  factors  are  definable  for  STARS 
and  that  defining  and  using  them  is  critical  to  the  success  of  the  Program.  These  goals,  objectives 
and  success  factors  should  be  established  at  designated  organizational  levels,  and  their  success  or 
failure  frequently  monitored. 

The  panel's  final  strategic  recommendation  is  that  STARS  management  gives  immediate 
attention  to  establishing  and  cultivating  customers,  advocates,  and  links  to  other  programs.  An 
important  supporting  tactic  is  to  move  toward  early,  demonstrably  beneficial,  results. 

6.0  CONCLUSION 

The  results  of  the  September-October  STARS  Program  Assessment  are  that  the  overall  goals 
of  STARS  are  important  and  sound  and  that  the  September  19  Plan  is  a  reasonable  starting  point  for 
further  development  of  the  Program.  There  are  serious,  but  solvable,  organizational  and 
management  problems.  A  new  emphasis  on  market  exploitation  and  the  alteration  of  DoD's  practices 
as  a  customer  for  software  should  be  established.  The  Program's  plans  should  be  executed  quickly 
so  that  early  results  will  occur.  Close  relationships  with  STARS  constituency  and  the  Program  s 
advocates  should  be  cultivated  as  soon  as  possible.  Finally,  a  review  and  reorientation  of  the 
technical  areas  and  teams  needs  to  be  done. 
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This  appendix  provides  a  detailed  explanation  of  the  STARS  Program's  constituencies  as  depicted  in 
Figure  7. 

A.  DCRBCRC 

The  DCRB  is  responsible  for  implementing  MCCR  acquisitions  policy,  and  as  such 
represents  the  management  level  personnel  directly  guiding  (and/or  enforcing)  use  of  STARS- 
generated  technology  and  policy.  This  board  also  is  the  top  level  group  capable  of  granting  waivers 
to  established  acquisition  policy. 

The  DCRB  is  shown,  in  Figure  7,  with  line  relationships  to  Computer  Systems  and  Software 
Directorate  (its  chairman),  and  to  the  JLC  Software  Initiative  who  has  a  representative  on  the  DCRB. 
The  DCRB  is  at  the  "top  of  the  figure"  because  of  its  heavy  representation  of  Assistant  Secretaries 
and  star-rank  officers. 

B  JLC  Software  Initiative 

This  initiative  area  is  most  widely  known  for  its  standards  producing  efforts  (MIL-STD’s 
originally  known  as  SDS  and  SQS,  now  registered  as  DoD-STDS-2 1 67  and  2168).  It  is  also 
engaged  in  producing  joint  service  policy,  business  practices,  methodology  (specifically  applied  to 
software  acquisiton),  measurement  (as  applied  to  software  quality),  and  management  practices 
Some  of  the  products  of  the  software  standards  project  include: 

•  Joint  Regulation  on  Computer  Resource  Management  in  Defense  Systems 

•  Software  Development  Standard  (DoD- STD-2 167) 

•  Software  Quality  (DoD-STD-2168) 

•  Configuration  Management  (MIL-STD-483A) 

•  Specification  Practices  (MIL-STD-490A) 

The  JLC  effort  attracted  industry  attention  and  participation  almost  as  intensively  as  the  Ada 
program  attracted  professional  and  academic  attention  and  participation.  The  industry-wide 
participation  continues  at  present  at  a  high  level;  the  September  1985  EIA  Annual  G-34  Panel 
Workshop  had  eight  large  panels  on  related  issues.  The  large  degree  of  participative  attention  had  a 
major  impact  on  the  development  of  the  Initiative's  products. 

The  products  have  a  strong  coupling  to  the  STARS  areas.  Particularly  in  technology- 
sensitive  areas,  such  as  Software  Practices.  Methodologies  and  Support  Environments,  synergism 
between  this  effort  and  STARS  would  be  beneficial  (particularly  with  the  already  in-place  working 
relationship  with  industry). 
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Interfaces  with  STARS  work  areas  have  been  suggested  by  JLC  Initiative  industry 
participants: 

(1)  SDSIssue5:  System  engineering  to  software  development  interface 
(Methodology,  SEE,  Business  Practices  Areas) 

(2)  SDS  Issue  8:  Ada  design  and  coding  standard  (Methodology  Area) 

(3)  SDS  Issue  13:  Design  Methodologies  (Methodology  Area) 

(4)  SDS  Issue  26:  Reusable  and  commercial  software  (Methodology,  Business 
Practices,  and  SEE  Areas) 

(5)  SDS  Issue  43:  Automation  of  MIL-STD-2 167  Documentation  (SEE  Area) 

(6)  SDS  Issues  56-60:  Artificial  Intelligence  software  development  in  MJL-STD-2 167 
context  (Applications  Specific  Area) 

The  "output”  of  the  JLC  Software  Initiative,  insofar  as  STARS  is  concerned,  is  technology  to 
be  transferred  to  the  constituent  community,  as  represented  in  the  figure.  There  is  no  predefined  path 
for  the  JLC  effort  to  receive  benefit  from  STARS  at  present 

C.  STARS  Joint  Program  Office 

1.  STARS  Area  teams 

The  STARS  Areas  presently  receive  direction  from  the  managing  DoD  components.  They  act 
to  manage  the  SJPO  area  effort  (presumably  contractor  produced)  that  will  provide  technology  to  be 
transferred  as  represented  in  the  figure. 

2  STARS  Area  contractors 

The  contractors  perform  the  work  required  to  meet  the  STARS  Area  goals,  as  directed 
primarily  by  the  STARS  Area  teams.  Being  members  of  industry,  these  constituents  are  also 
members  of  their  individual  employing  organizations  (who  review  the  workers),  and  the  employers 
are  members  of  respective  industry  associations.  The  workers  are  likely  to  also  be  involved  with 
professional  societies  such  as  IEEE  and  ACM  and  thus  be  influenced  by  colleagues  (on  a  personal 
basis)  as  well  as  management  (cooperatively  interfacing  with  management  of  other  industrial 
concents). 

The  contractors,  furthermore,  derive  the  basis  for  their  technology  from  industry  (and 
academia).  They  incorporate  ideas  from  their  staffs  and  IR&D  efforts,  and  integrate  them  with  ideas 
from  published  sources,  trade  sources,  and  staff  colleagues. 

3.  STARS/Industry  "Joint  Working  Group" 

Joint  SJPO/Industry  efforts  have  occurred  in  working  groups,  such  as  the  1985  NSLA  San 
Diego  Conference,  and  the  1984  and  1985  EIA  Annual  Workshop  SEE  Area  panels.  These 
represent  non-sponsored  opportunities  to  interact,  both  on  a  technology  development  plan  and  on  a 
management  and  practices  plan.  (This  topic  does  not  refer  to  the  STARS  sponsored  workshops, 
which  are  listed  under  "Technology  Transfer  Constituents".) 


D.  Technology  Transfer  Constituencies 


This  category  includes  interfacing  parties  which  derive  benefit  from  the  STARS,  Ada  JLC 
Software  Initiative,  and  Industry/ Academia  Programs. 

1  Software  Engineering  Institute 

The  SEI  is  chartered  to  transfer  technology  developed  by  the  STARS  program  to  Industrv 
and  the  Services.  One  mechanism  which  will  be  used  by  the  SEI  is  their  "Industry  Affiliates" 
program,  wherein  Industry  members  serve  in  temporary  staff  positions  at  the  SEI. 

2  DoD/Industry  Workshops 

STARS  sponsored  joint  workshops  function  as  an  interface  point  where  STARS  technology 
is  disclosed  to  industry. 

3  Conferences 

Conferences  are  sponsored  by  several  groups:  the  Industry  Associations  and  the  DoD 
respectively  sponsored  portions  of  the  "San  Diego"  conferences  in  Spring  1985,  and  professional 
groups  such  as  IEEE  and  ACM  sponsor  conferences  with  relevant  sessions. 

4.  Technology  Development  Constituencies 

This  category  includes  constituents  which  develop  technology  to  be  incorporated  by  STARS 
and  the  STARS  contractors. 

( 1 )  DoD  Working  Groups 

(2)  DoD  Industry  Workshops 

(3)  Industry  Working  Groups 

(4)  Industry  and  Professional  Organizations 

(5)  Industry 

(6)  Industry  (Private)  Consortia 

(7)  Academia 

(8)  DoD  Component  Laboratories 

(9)  DoD  Defense  Programs 

5.  Standards  Community  Constituents 
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6.  DoD  Constituents 

( 1 )  Service  entities  participating  in  STARS 

(2)  Service  entity  consumers  who  might  be  STARS  technology  users 

7.  ADA  Joint  Program  Office 

This  office  sponsors  five  activities  which  provide  technology  which  will  be  transferred  to 
external  constituents  (and  to  the  STARS  program): 

(1)  KIT/KITIA:  These  organizations  have  defined  a  set  of  tool  builder  (environment  user 
interfaces  to  kernel  services. 

(2)  Ada  Contractors:  These  entities  are  providing  Ada  compilers,  tool  products,  and 
environment  products. 

(3)  Industry  Ada  Programs:  A  frequent  contributor  to  the  technology  base  for  software  is 
software  IR&D.  These  funds  are  often  used  to  develop  company-specific  tools  and  tool 
sets  for  future  use  on  projects  for  the  internal  DoD  (constituent)  users. 

(4)  Ada  Validation  Facility:  This  facility's  greatest  contribution  is  in  the  stabilization  of  the 
Ada  language,  and  the  application  of  such  stabilizing  technology  to  other  areas  (i.e., 
environment  and  tool  standard  interfaces). 
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